2023-04-13 20:53:31,950 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24 2023-04-13 20:53:31,963 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-13 20:53:31,980 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-13 20:53:31,981 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582, deleteOnExit=true 2023-04-13 20:53:31,981 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-13 20:53:31,982 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/test.cache.data in system properties and HBase conf 2023-04-13 20:53:31,982 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/hadoop.tmp.dir in system properties and HBase conf 2023-04-13 20:53:31,983 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/hadoop.log.dir in system properties and HBase conf 2023-04-13 20:53:31,983 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-13 20:53:31,984 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-13 20:53:31,984 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-13 20:53:32,099 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-13 20:53:32,522 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-13 20:53:32,527 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-13 20:53:32,528 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-13 20:53:32,528 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-13 20:53:32,529 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 20:53:32,529 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-13 20:53:32,530 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-13 20:53:32,530 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-13 20:53:32,530 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 20:53:32,531 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-13 20:53:32,531 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/nfs.dump.dir in system properties and HBase conf 2023-04-13 20:53:32,532 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/java.io.tmpdir in system properties and HBase conf 2023-04-13 20:53:32,532 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-13 20:53:32,532 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-13 20:53:32,533 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-13 20:53:33,158 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 20:53:33,163 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 20:53:33,476 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-13 20:53:33,648 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-13 20:53:33,669 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 20:53:33,717 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 20:53:33,748 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/java.io.tmpdir/Jetty_localhost_38927_hdfs____9gmpur/webapp 2023-04-13 20:53:33,889 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38927 2023-04-13 20:53:33,927 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-13 20:53:33,928 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-13 20:53:34,356 WARN [Listener at localhost/39631] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 20:53:34,493 WARN [Listener at localhost/39631] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-13 20:53:34,512 WARN [Listener at localhost/39631] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-13 20:53:34,519 INFO [Listener at localhost/39631] log.Slf4jLog(67): jetty-6.1.26 2023-04-13 20:53:34,525 INFO [Listener at localhost/39631] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/java.io.tmpdir/Jetty_localhost_37191_datanode____ijpuh8/webapp 2023-04-13 20:53:34,652 INFO [Listener at localhost/39631] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37191 2023-04-13 20:53:35,060 WARN [Listener at localhost/42015] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-13 20:53:35,673 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x38bb7e023c1766d0: Processing first storage report for DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4 from datanode 1651cc58-dcf5-4483-aee0-3033ee4713c7 2023-04-13 20:53:35,675 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x38bb7e023c1766d0: from storage DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4 node DatanodeRegistration(127.0.0.1:40511, datanodeUuid=1651cc58-dcf5-4483-aee0-3033ee4713c7, infoPort=46499, infoSecurePort=0, ipcPort=42015, storageInfo=lv=-57;cid=testClusterID;nsid=302171585;c=1681419213238), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-13 20:53:35,675 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x38bb7e023c1766d0: Processing first storage report for DS-ef5327e9-a78a-4e1f-b719-f4d244f7bc24 from datanode 1651cc58-dcf5-4483-aee0-3033ee4713c7 2023-04-13 20:53:35,675 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x38bb7e023c1766d0: from storage DS-ef5327e9-a78a-4e1f-b719-f4d244f7bc24 node DatanodeRegistration(127.0.0.1:40511, datanodeUuid=1651cc58-dcf5-4483-aee0-3033ee4713c7, infoPort=46499, infoSecurePort=0, ipcPort=42015, storageInfo=lv=-57;cid=testClusterID;nsid=302171585;c=1681419213238), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-13 20:53:35,759 DEBUG [Listener at localhost/42015] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24 2023-04-13 20:53:35,820 INFO [Listener at localhost/42015] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582/zookeeper_0, clientPort=54584, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-13 20:53:35,834 INFO [Listener at localhost/42015] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54584 2023-04-13 20:53:35,843 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:35,845 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:36,507 INFO [Listener at localhost/42015] util.FSUtils(471): Created version file at hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe with version=8 2023-04-13 20:53:36,507 INFO [Listener at localhost/42015] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/hbase-staging 2023-04-13 20:53:36,827 INFO [Listener at localhost/42015] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-13 20:53:37,287 INFO [Listener at localhost/42015] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 20:53:37,322 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,323 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,323 INFO [Listener at localhost/42015] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 20:53:37,323 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,323 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 20:53:37,486 INFO [Listener at localhost/42015] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 20:53:37,557 DEBUG [Listener at localhost/42015] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-13 20:53:37,654 INFO [Listener at localhost/42015] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43163 2023-04-13 20:53:37,664 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:37,666 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:37,686 INFO [Listener at localhost/42015] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43163 connecting to ZooKeeper ensemble=127.0.0.1:54584 2023-04-13 20:53:37,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:431630x0, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 20:53:37,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43163-0x1017f4d0d830000 connected 2023-04-13 20:53:37,764 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 20:53:37,765 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 20:53:37,768 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 20:53:37,776 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43163 2023-04-13 20:53:37,777 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43163 2023-04-13 20:53:37,777 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43163 2023-04-13 20:53:37,777 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43163 2023-04-13 20:53:37,777 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43163 2023-04-13 20:53:37,784 INFO [Listener at localhost/42015] master.HMaster(444): hbase.rootdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe, hbase.cluster.distributed=false 2023-04-13 20:53:37,857 INFO [Listener at localhost/42015] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-13 20:53:37,858 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,858 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,858 INFO [Listener at localhost/42015] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-13 20:53:37,858 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-13 20:53:37,858 INFO [Listener at localhost/42015] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-13 20:53:37,863 INFO [Listener at localhost/42015] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-13 20:53:37,866 INFO [Listener at localhost/42015] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36431 2023-04-13 20:53:37,868 INFO [Listener at localhost/42015] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-13 20:53:37,874 DEBUG [Listener at localhost/42015] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-13 20:53:37,875 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:37,878 INFO [Listener at localhost/42015] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:37,880 INFO [Listener at localhost/42015] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36431 connecting to ZooKeeper ensemble=127.0.0.1:54584 2023-04-13 20:53:37,884 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:364310x0, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-13 20:53:37,885 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): regionserver:364310x0, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 20:53:37,889 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): regionserver:364310x0, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 20:53:37,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36431-0x1017f4d0d830001 connected 2023-04-13 20:53:37,891 DEBUG [Listener at localhost/42015] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-13 20:53:37,892 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36431 2023-04-13 20:53:37,892 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36431 2023-04-13 20:53:37,893 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36431 2023-04-13 20:53:37,893 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36431 2023-04-13 20:53:37,894 DEBUG [Listener at localhost/42015] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36431 2023-04-13 20:53:37,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:37,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 20:53:37,906 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:37,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 20:53:37,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-13 20:53:37,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:37,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 20:53:37,928 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43163,1681419216653 from backup master directory 2023-04-13 20:53:37,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-13 20:53:37,931 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:37,931 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-13 20:53:37,932 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 20:53:37,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:37,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-13 20:53:37,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-13 20:53:38,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/hbase.id with ID: bc736819-3abb-4ecf-9281-1ea88f3874ff 2023-04-13 20:53:38,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-13 20:53:38,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:38,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5ded93 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:38,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1b96126e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:38,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:53:38,199 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-13 20:53:38,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-13 20:53:38,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-13 20:53:38,219 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:38,223 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:38,224 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 20:53:38,258 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store-tmp 2023-04-13 20:53:38,288 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:38,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 20:53:38,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:53:38,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:53:38,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 20:53:38,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:53:38,289 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:53:38,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 20:53:38,291 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/WALs/jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:38,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43163%2C1681419216653, suffix=, logDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/WALs/jenkins-hbase4.apache.org,43163,1681419216653, archiveDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/oldWALs, maxLogs=10 2023-04-13 20:53:38,375 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK] 2023-04-13 20:53:38,383 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:38,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/WALs/jenkins-hbase4.apache.org,43163,1681419216653/jenkins-hbase4.apache.org%2C43163%2C1681419216653.1681419218329 2023-04-13 20:53:38,443 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK]] 2023-04-13 20:53:38,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:53:38,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:38,448 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,510 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,517 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-13 20:53:38,542 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-13 20:53:38,555 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:38,562 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,564 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-13 20:53:38,592 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:53:38,594 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60227029, jitterRate=-0.10254733264446259}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 20:53:38,594 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 20:53:38,596 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-13 20:53:38,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-13 20:53:38,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-13 20:53:38,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-13 20:53:38,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-13 20:53:38,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 35 msec 2023-04-13 20:53:38,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-13 20:53:38,689 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-13 20:53:38,694 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-13 20:53:38,726 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-13 20:53:38,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-13 20:53:38,733 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-13 20:53:38,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-13 20:53:38,746 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-13 20:53:38,750 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:38,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-13 20:53:38,754 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-13 20:53:38,769 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-13 20:53:38,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 20:53:38,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-13 20:53:38,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:38,775 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43163,1681419216653, sessionid=0x1017f4d0d830000, setting cluster-up flag (Was=false) 2023-04-13 20:53:38,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:38,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-13 20:53:38,801 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:38,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:38,816 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-13 20:53:38,817 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:38,820 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.hbase-snapshot/.tmp 2023-04-13 20:53:38,898 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(952): ClusterId : bc736819-3abb-4ecf-9281-1ea88f3874ff 2023-04-13 20:53:38,902 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-13 20:53:38,908 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-13 20:53:38,908 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-13 20:53:38,912 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-13 20:53:38,913 DEBUG [RS:0;jenkins-hbase4:36431] zookeeper.ReadOnlyZKClient(139): Connect 0x4c4735ad to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:38,923 DEBUG [RS:0;jenkins-hbase4:36431] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4280c255, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:38,924 DEBUG [RS:0;jenkins-hbase4:36431] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46bf5edd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 20:53:38,942 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-13 20:53:38,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 20:53:38,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 20:53:38,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:38,954 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681419248954 2023-04-13 20:53:38,956 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-13 20:53:38,959 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36431 2023-04-13 20:53:38,961 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 20:53:38,961 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-13 20:53:38,964 INFO [RS:0;jenkins-hbase4:36431] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-13 20:53:38,964 INFO [RS:0;jenkins-hbase4:36431] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-13 20:53:38,964 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1023): About to register with Master. 2023-04-13 20:53:38,967 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 20:53:38,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-13 20:53:38,970 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43163,1681419216653 with isa=jenkins-hbase4.apache.org/172.31.14.131:36431, startcode=1681419217857 2023-04-13 20:53:38,977 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-13 20:53:38,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-13 20:53:38,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-13 20:53:38,978 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-13 20:53:38,980 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:38,982 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-13 20:53:38,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-13 20:53:38,983 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-13 20:53:38,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-13 20:53:38,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-13 20:53:38,987 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681419218987,5,FailOnTimeoutGroup] 2023-04-13 20:53:38,988 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681419218987,5,FailOnTimeoutGroup] 2023-04-13 20:53:38,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:38,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-13 20:53:38,989 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:38,990 DEBUG [RS:0;jenkins-hbase4:36431] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-13 20:53:38,990 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,006 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 20:53:39,007 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-13 20:53:39,008 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe 2023-04-13 20:53:39,030 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:39,033 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 20:53:39,037 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/info 2023-04-13 20:53:39,038 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 20:53:39,039 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,040 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 20:53:39,043 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/rep_barrier 2023-04-13 20:53:39,044 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 20:53:39,046 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,047 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 20:53:39,049 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/table 2023-04-13 20:53:39,050 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 20:53:39,051 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,053 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740 2023-04-13 20:53:39,054 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740 2023-04-13 20:53:39,056 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41955, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-13 20:53:39,058 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:53:39,061 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 20:53:39,067 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:53:39,068 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64882613, jitterRate=-0.03317372500896454}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:53:39,068 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 20:53:39,068 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 20:53:39,068 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 20:53:39,068 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 20:53:39,068 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 20:53:39,068 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 20:53:39,069 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 20:53:39,070 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 20:53:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,075 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-13 20:53:39,076 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-13 20:53:39,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-13 20:53:39,088 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe 2023-04-13 20:53:39,088 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:39631 2023-04-13 20:53:39,088 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-13 20:53:39,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 20:53:39,094 DEBUG [RS:0;jenkins-hbase4:36431] zookeeper.ZKUtil(162): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,094 WARN [RS:0;jenkins-hbase4:36431] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-13 20:53:39,095 INFO [RS:0;jenkins-hbase4:36431] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 20:53:39,095 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1947): logDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/WALs/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,096 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36431,1681419217857] 2023-04-13 20:53:39,108 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-13 20:53:39,111 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-13 20:53:39,112 DEBUG [RS:0;jenkins-hbase4:36431] zookeeper.ZKUtil(162): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,123 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-13 20:53:39,133 INFO [RS:0;jenkins-hbase4:36431] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-13 20:53:39,159 INFO [RS:0;jenkins-hbase4:36431] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-13 20:53:39,163 INFO [RS:0;jenkins-hbase4:36431] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-13 20:53:39,164 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,165 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-13 20:53:39,173 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,173 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,174 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,175 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,175 DEBUG [RS:0;jenkins-hbase4:36431] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-13 20:53:39,177 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,177 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,177 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,195 INFO [RS:0;jenkins-hbase4:36431] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-13 20:53:39,197 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36431,1681419217857-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,222 INFO [RS:0;jenkins-hbase4:36431] regionserver.Replication(203): jenkins-hbase4.apache.org,36431,1681419217857 started 2023-04-13 20:53:39,222 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36431,1681419217857, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36431, sessionid=0x1017f4d0d830001 2023-04-13 20:53:39,223 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-13 20:53:39,223 DEBUG [RS:0;jenkins-hbase4:36431] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,223 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36431,1681419217857' 2023-04-13 20:53:39,223 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:39,226 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36431,1681419217857' 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-13 20:53:39,227 DEBUG [RS:0;jenkins-hbase4:36431] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-13 20:53:39,228 DEBUG [RS:0;jenkins-hbase4:36431] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-13 20:53:39,228 INFO [RS:0;jenkins-hbase4:36431] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-13 20:53:39,228 INFO [RS:0;jenkins-hbase4:36431] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-13 20:53:39,264 DEBUG [jenkins-hbase4:43163] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-13 20:53:39,267 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36431,1681419217857, state=OPENING 2023-04-13 20:53:39,276 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-13 20:53:39,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:39,278 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 20:53:39,282 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:53:39,341 INFO [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36431%2C1681419217857, suffix=, logDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/WALs/jenkins-hbase4.apache.org,36431,1681419217857, archiveDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/oldWALs, maxLogs=32 2023-04-13 20:53:39,361 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK] 2023-04-13 20:53:39,365 INFO [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/WALs/jenkins-hbase4.apache.org,36431,1681419217857/jenkins-hbase4.apache.org%2C36431%2C1681419217857.1681419219344 2023-04-13 20:53:39,366 DEBUG [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK]] 2023-04-13 20:53:39,472 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:39,475 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-13 20:53:39,479 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42304, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-13 20:53:39,492 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-13 20:53:39,492 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-13 20:53:39,496 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36431%2C1681419217857.meta, suffix=.meta, logDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/WALs/jenkins-hbase4.apache.org,36431,1681419217857, archiveDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/oldWALs, maxLogs=32 2023-04-13 20:53:39,516 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK] 2023-04-13 20:53:39,521 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/WALs/jenkins-hbase4.apache.org,36431,1681419217857/jenkins-hbase4.apache.org%2C36431%2C1681419217857.meta.1681419219498.meta 2023-04-13 20:53:39,521 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40511,DS-e5d3c178-663f-4e15-ae17-74c2e94ad1f4,DISK]] 2023-04-13 20:53:39,522 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:53:39,523 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-13 20:53:39,542 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-13 20:53:39,547 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-13 20:53:39,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-13 20:53:39,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:39,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-13 20:53:39,553 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-13 20:53:39,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-13 20:53:39,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/info 2023-04-13 20:53:39,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/info 2023-04-13 20:53:39,561 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-13 20:53:39,562 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,562 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-13 20:53:39,564 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/rep_barrier 2023-04-13 20:53:39,564 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/rep_barrier 2023-04-13 20:53:39,565 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-13 20:53:39,566 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,566 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-13 20:53:39,568 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/table 2023-04-13 20:53:39,568 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/table 2023-04-13 20:53:39,569 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-13 20:53:39,572 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:39,575 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740 2023-04-13 20:53:39,577 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740 2023-04-13 20:53:39,582 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:53:39,585 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-13 20:53:39,587 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67834324, jitterRate=0.01081019639968872}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:53:39,587 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-13 20:53:39,600 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681419219464 2023-04-13 20:53:39,626 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-13 20:53:39,627 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-13 20:53:39,627 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36431,1681419217857, state=OPEN 2023-04-13 20:53:39,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-13 20:53:39,630 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-13 20:53:39,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-13 20:53:39,636 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36431,1681419217857 in 348 msec 2023-04-13 20:53:39,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-13 20:53:39,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 551 msec 2023-04-13 20:53:39,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 780 msec 2023-04-13 20:53:39,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681419219647, completionTime=-1 2023-04-13 20:53:39,647 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-13 20:53:39,648 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-13 20:53:39,711 DEBUG [hconnection-0x2c6dc074-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:39,714 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:39,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-13 20:53:39,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681419279730 2023-04-13 20:53:39,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681419339730 2023-04-13 20:53:39,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 82 msec 2023-04-13 20:53:39,757 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43163,1681419216653-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43163,1681419216653-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43163,1681419216653-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43163, period=300000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-13 20:53:39,768 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-13 20:53:39,778 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-13 20:53:39,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-13 20:53:39,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-13 20:53:39,796 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:53:39,799 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:53:39,828 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:39,831 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674 empty. 2023-04-13 20:53:39,832 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:39,832 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-13 20:53:39,858 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-13 20:53:39,860 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 33a8ac462e5249c56ee9b85ce2d17674, NAME => 'hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:53:39,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:39,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 33a8ac462e5249c56ee9b85ce2d17674, disabling compactions & flushes 2023-04-13 20:53:39,878 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:39,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:39,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. after waiting 0 ms 2023-04-13 20:53:39,878 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:39,878 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:39,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 33a8ac462e5249c56ee9b85ce2d17674: 2023-04-13 20:53:39,884 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:53:39,900 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681419219887"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419219887"}]},"ts":"1681419219887"} 2023-04-13 20:53:39,930 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:53:39,932 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:53:39,939 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419219932"}]},"ts":"1681419219932"} 2023-04-13 20:53:39,943 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-13 20:53:39,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=33a8ac462e5249c56ee9b85ce2d17674, ASSIGN}] 2023-04-13 20:53:39,956 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=33a8ac462e5249c56ee9b85ce2d17674, ASSIGN 2023-04-13 20:53:39,958 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=33a8ac462e5249c56ee9b85ce2d17674, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:53:40,109 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=33a8ac462e5249c56ee9b85ce2d17674, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:40,109 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681419220108"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419220108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419220108"}]},"ts":"1681419220108"} 2023-04-13 20:53:40,114 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 33a8ac462e5249c56ee9b85ce2d17674, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:53:40,276 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:40,277 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 33a8ac462e5249c56ee9b85ce2d17674, NAME => 'hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:53:40,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:40,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,279 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,281 INFO [StoreOpener-33a8ac462e5249c56ee9b85ce2d17674-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,283 DEBUG [StoreOpener-33a8ac462e5249c56ee9b85ce2d17674-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/info 2023-04-13 20:53:40,284 DEBUG [StoreOpener-33a8ac462e5249c56ee9b85ce2d17674-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/info 2023-04-13 20:53:40,284 INFO [StoreOpener-33a8ac462e5249c56ee9b85ce2d17674-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 33a8ac462e5249c56ee9b85ce2d17674 columnFamilyName info 2023-04-13 20:53:40,285 INFO [StoreOpener-33a8ac462e5249c56ee9b85ce2d17674-1] regionserver.HStore(310): Store=33a8ac462e5249c56ee9b85ce2d17674/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:40,287 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,287 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:53:40,297 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:53:40,298 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 33a8ac462e5249c56ee9b85ce2d17674; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59291366, jitterRate=-0.11648979783058167}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-13 20:53:40,298 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 33a8ac462e5249c56ee9b85ce2d17674: 2023-04-13 20:53:40,300 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674., pid=6, masterSystemTime=1681419220268 2023-04-13 20:53:40,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:40,305 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:53:40,306 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=33a8ac462e5249c56ee9b85ce2d17674, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:40,306 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681419220305"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419220305"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419220305"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419220305"}]},"ts":"1681419220305"} 2023-04-13 20:53:40,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-13 20:53:40,316 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 33a8ac462e5249c56ee9b85ce2d17674, server=jenkins-hbase4.apache.org,36431,1681419217857 in 198 msec 2023-04-13 20:53:40,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-13 20:53:40,321 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=33a8ac462e5249c56ee9b85ce2d17674, ASSIGN in 363 msec 2023-04-13 20:53:40,322 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:53:40,322 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419220322"}]},"ts":"1681419220322"} 2023-04-13 20:53:40,326 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-13 20:53:40,332 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:53:40,335 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 551 msec 2023-04-13 20:53:40,396 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-13 20:53:40,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-13 20:53:40,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:53:40,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-13 20:53:40,447 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 20:53:40,454 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-04-13 20:53:40,466 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-13 20:53:40,484 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-13 20:53:40,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 22 msec 2023-04-13 20:53:40,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-13 20:53:40,506 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-13 20:53:40,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.574sec 2023-04-13 20:53:40,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-13 20:53:40,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-13 20:53:40,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-13 20:53:40,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43163,1681419216653-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-13 20:53:40,511 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43163,1681419216653-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-13 20:53:40,521 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-13 20:53:40,604 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x373f9c99 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:40,609 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6b116c72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:40,622 DEBUG [hconnection-0x25e7ed72-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:40,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42312, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:40,644 INFO [Listener at localhost/42015] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:53:40,666 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=323, ProcessCount=170, AvailableMemoryMB=1497 2023-04-13 20:53:40,684 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:53:40,688 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43462, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:53:40,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:53:40,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:53:40,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:53:40,710 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:53:40,713 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:53:40,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-13 20:53:40,717 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:40,718 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb empty. 2023-04-13 20:53:40,720 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:40,720 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:53:40,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 20:53:40,738 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:53:40,740 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => cdf6485ca98235662ec10d4d33684acb, NAME => 'TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:53:40,751 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:40,752 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing cdf6485ca98235662ec10d4d33684acb, disabling compactions & flushes 2023-04-13 20:53:40,752 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:40,752 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:40,752 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. after waiting 0 ms 2023-04-13 20:53:40,752 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:40,752 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:40,752 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:40,756 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:53:40,757 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419220757"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419220757"}]},"ts":"1681419220757"} 2023-04-13 20:53:40,760 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:53:40,762 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:53:40,762 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419220762"}]},"ts":"1681419220762"} 2023-04-13 20:53:40,764 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:53:40,770 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, ASSIGN}] 2023-04-13 20:53:40,784 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, ASSIGN 2023-04-13 20:53:40,786 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:53:40,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 20:53:40,937 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cdf6485ca98235662ec10d4d33684acb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:40,938 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419220937"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419220937"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419220937"}]},"ts":"1681419220937"} 2023-04-13 20:53:40,941 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:53:41,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 20:53:41,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cdf6485ca98235662ec10d4d33684acb, NAME => 'TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:53:41,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:53:41,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,105 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,107 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A 2023-04-13 20:53:41,107 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A 2023-04-13 20:53:41,109 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:53:41,109 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cdf6485ca98235662ec10d4d33684acb columnFamilyName A 2023-04-13 20:53:41,110 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(310): Store=cdf6485ca98235662ec10d4d33684acb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:41,110 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,112 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B 2023-04-13 20:53:41,112 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B 2023-04-13 20:53:41,113 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:53:41,113 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cdf6485ca98235662ec10d4d33684acb columnFamilyName B 2023-04-13 20:53:41,114 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(310): Store=cdf6485ca98235662ec10d4d33684acb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:41,114 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,116 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C 2023-04-13 20:53:41,116 DEBUG [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C 2023-04-13 20:53:41,116 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:53:41,117 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cdf6485ca98235662ec10d4d33684acb columnFamilyName C 2023-04-13 20:53:41,117 INFO [StoreOpener-cdf6485ca98235662ec10d4d33684acb-1] regionserver.HStore(310): Store=cdf6485ca98235662ec10d4d33684acb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:53:41,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:53:41,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:53:41,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cdf6485ca98235662ec10d4d33684acb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68432650, jitterRate=0.019725948572158813}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:53:41,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:41,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., pid=11, masterSystemTime=1681419221094 2023-04-13 20:53:41,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,166 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cdf6485ca98235662ec10d4d33684acb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,167 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419221166"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419221166"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419221166"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419221166"}]},"ts":"1681419221166"} 2023-04-13 20:53:41,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-13 20:53:41,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 228 msec 2023-04-13 20:53:41,175 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-13 20:53:41,176 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, ASSIGN in 402 msec 2023-04-13 20:53:41,177 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:53:41,177 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419221177"}]},"ts":"1681419221177"} 2023-04-13 20:53:41,179 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:53:41,182 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:53:41,184 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 477 msec 2023-04-13 20:53:41,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-13 20:53:41,339 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-13 20:53:41,343 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x35594b1b to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,348 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55e5fd59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,351 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,353 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,357 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:53:41,359 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43466, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:53:41,369 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,373 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56223d42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,373 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4fea27db to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,378 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,379 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,385 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de4d1f6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,385 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x63859c62 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,389 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,390 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x42927198 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,393 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60b8c4c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,394 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,400 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,401 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x3cd3abbd to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,428 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6129209e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,429 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,433 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,433 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4808c6d5 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:53:41,437 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1423ce11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:53:41,462 DEBUG [hconnection-0x106b4c0f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,464 DEBUG [hconnection-0x2393f3c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,465 DEBUG [hconnection-0x5b12d690-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,466 DEBUG [hconnection-0x71b63d66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,469 DEBUG [hconnection-0x39df0911-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,470 DEBUG [hconnection-0x30bca569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,470 DEBUG [hconnection-0x30cf85db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,470 DEBUG [hconnection-0x2035e7e9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,470 DEBUG [hconnection-0x1da1ae0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:53:41,475 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42342, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,482 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42330, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,482 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,483 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,483 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,487 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42364, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,487 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42394, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,496 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,496 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:53:41,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:41,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:41,530 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:41,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:41,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:41,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:41,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,538 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:41,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:41,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:41,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:41,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:41,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:41,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:41,587 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:41,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:53:41,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:41,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:41,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,593 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:41,593 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:41,595 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:41,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:41,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:41,601 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,601 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,602 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:41,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:41,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:41,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffc081b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:41,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:41,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:41,610 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:41,611 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:41,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:41,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:41,618 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:41,615 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,624 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:41,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:41,627 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:41,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:41,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:41,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:41,627 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:41,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:41,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:41,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:41,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:41,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:41,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:41,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:41,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:41,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:41,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:41,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,675 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:41,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:41,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:41,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419281710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419281713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419281714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419281714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419281718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:41,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:41,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/49c1e436865b4c4eb508c1ead8a24488 2023-04-13 20:53:41,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:41,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:41,804 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:41,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:41,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:41,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:41,809 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,809 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:41,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:41,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:41,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:41,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:41,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:41,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:41,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:41,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:41,828 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:41,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:41,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:41,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:41,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31dc2125[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:41,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:41,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,857 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:41,857 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:41,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:41,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419281856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:41,861 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:41,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419281856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419281861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419281863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:41,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:41,874 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:41,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:41,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:41,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:41,874 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:41,876 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:41,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419281865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:41,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:41,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:41,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:41,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:41,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:41,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:41,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:41,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:41,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:41,903 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:41,903 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:41,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:41,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:41,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,003 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:42,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:42,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cdba5971b81641f09dbb1d3fe855c48e 2023-04-13 20:53:42,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:42,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:42,020 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:42,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:42,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:42,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,023 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:42,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:42,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:42,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:42,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:42,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:42,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:42,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:42,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:42,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:42,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:42,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:42,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639b1c78[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:42,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:42,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,066 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,066 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:42,066 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:42,066 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:42,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:42,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419282069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419282069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419282066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,077 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:42,085 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:42,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:42,085 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:42,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419282083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419282090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:42,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,109 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,109 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,109 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:42,109 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:42,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/a21d5e10fb044536b389c1ba26daa828 2023-04-13 20:53:42,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:42,126 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/49c1e436865b4c4eb508c1ead8a24488 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488 2023-04-13 20:53:42,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 20:53:42,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cdba5971b81641f09dbb1d3fe855c48e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e 2023-04-13 20:53:42,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 20:53:42,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/a21d5e10fb044536b389c1ba26daa828 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828 2023-04-13 20:53:42,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828, entries=150, sequenceid=15, filesize=11.6 K 2023-04-13 20:53:42,198 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cdf6485ca98235662ec10d4d33684acb in 609ms, sequenceid=15, compaction requested=false 2023-04-13 20:53:42,200 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 20:53:42,207 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:42,209 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:42,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:42,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:42,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:42,223 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:42,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:42,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:42,231 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,231 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,234 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:42,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:42,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:42,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:42,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:42,237 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:42,241 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:42,246 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,248 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:42,249 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:53:42,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:42,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:42,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:42,250 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,373 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/af6566682db34fc88f98d68a9d721e11 2023-04-13 20:53:42,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:42,379 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:42,435 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/f47be67d373d49daaedcad81e51fbef6 2023-04-13 20:53:42,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,525 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/75c3022945be471697fef4c24035d6e4 2023-04-13 20:53:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419282493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419282500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419282500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419282504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419282516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,551 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/af6566682db34fc88f98d68a9d721e11 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11 2023-04-13 20:53:42,562 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11, entries=150, sequenceid=33, filesize=11.6 K 2023-04-13 20:53:42,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/f47be67d373d49daaedcad81e51fbef6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6 2023-04-13 20:53:42,583 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6, entries=150, sequenceid=33, filesize=11.6 K 2023-04-13 20:53:42,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/75c3022945be471697fef4c24035d6e4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4 2023-04-13 20:53:42,604 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4, entries=150, sequenceid=33, filesize=11.6 K 2023-04-13 20:53:42,613 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=87.22 KB/89310 for cdf6485ca98235662ec10d4d33684acb in 365ms, sequenceid=33, compaction requested=false 2023-04-13 20:53:42,614 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:42,614 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,614 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:42,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:42,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:42,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:42,618 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:42,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44ed34da[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:42,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:42,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,625 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:42,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:42,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:42,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:42,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:42,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:42,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:42,638 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:42,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:42,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:42,639 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:42,639 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:42,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,643 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:53:42,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:42,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:42,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:42,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:42,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:42,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,671 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:42,671 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:42,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:42,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/ade28c1ba2084d3ab952d124de3abbe3 2023-04-13 20:53:42,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419282693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419282713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,726 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419282718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419282718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419282718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,771 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:42,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:42,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:42,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:42,809 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:42,809 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:42,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:42,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,818 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/59a3d15eedbf4178bbd9edb5aa6c4a83 2023-04-13 20:53:42,818 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,819 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:42,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:42,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:42,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:42,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:42,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:42,826 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,826 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:42,826 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:42,827 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:42,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:42,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:42,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:42,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419282831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:42,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9e508c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:42,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:42,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419282832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419282834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419282832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:42,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:42,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:42,842 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:42,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:42,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:42,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419282832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:42,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:42,852 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:42,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:42,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:42,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:42,852 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:42,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:42,858 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:42,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:42,871 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:42,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,872 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:42,872 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:42,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:42,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:42,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:42,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:42,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:42,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:42,986 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:42,986 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:42,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:42,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,991 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,991 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:42,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:42,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:42,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:42,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:42,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:42,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:42,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:42,993 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:42,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:43,000 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:43,000 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:43,000 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:43,000 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:43,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,006 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:43,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1b9dd3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:43,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:43,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:43,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:43,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:43,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:43,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:43,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:43,026 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:43,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:43,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:43,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:43,026 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:43,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:43,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,039 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:43,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:43,047 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419283044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,040 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419283044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419283045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419283045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419283052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:43,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:43,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:43,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:43,158 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:43,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:43,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:43,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:43,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:43,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:43,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:43,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:43,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:43,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:43,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:43,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:43,181 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,181 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:43,181 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:43,181 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:43,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:43,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:43,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d82214[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:43,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:43,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,190 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:43,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:43,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:43,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:43,198 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:43,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:43,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:43,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:43,198 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:43,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,214 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,214 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:43,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:43,214 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:43,217 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:43,217 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:43,311 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/def0aea8f60d4b319daa37701c1ab872 2023-04-13 20:53:43,317 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:43,322 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/ade28c1ba2084d3ab952d124de3abbe3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3 2023-04-13 20:53:43,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:43,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:43,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:43,327 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:43,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,328 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:43,328 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:43,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,330 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,331 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,331 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:43,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:43,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:43,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:43,336 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:43,336 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 20:53:43,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:43,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:43,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:43,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:43,338 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:43,338 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,338 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:43,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:43,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:43,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:43,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/59a3d15eedbf4178bbd9edb5aa6c4a83 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83 2023-04-13 20:53:43,343 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,343 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:43,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:43,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbe8131[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:43,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:43,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:43,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:43,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:43,353 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,353 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:43,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419283354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,356 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 20:53:43,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419283354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/def0aea8f60d4b319daa37701c1ab872 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872 2023-04-13 20:53:43,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419283355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419283361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:43,369 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:43,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:43,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:43,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:43,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419283361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:43,372 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:43,374 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,374 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,374 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872, entries=150, sequenceid=51, filesize=11.6 K 2023-04-13 20:53:43,386 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for cdf6485ca98235662ec10d4d33684acb in 742ms, sequenceid=51, compaction requested=true 2023-04-13 20:53:43,386 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:43,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:43,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:43,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:43,389 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:43,389 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:43,389 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,389 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,389 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,389 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,389 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:43,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:43,393 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:43,393 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:43,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,395 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:43,397 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:43,397 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,397 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=34.8 K 2023-04-13 20:53:43,399 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:43,399 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:43,399 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,399 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=34.8 K 2023-04-13 20:53:43,399 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cdba5971b81641f09dbb1d3fe855c48e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681419221507 2023-04-13 20:53:43,401 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f47be67d373d49daaedcad81e51fbef6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681419221676 2023-04-13 20:53:43,402 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 59a3d15eedbf4178bbd9edb5aa6c4a83, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:43,408 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 49c1e436865b4c4eb508c1ead8a24488, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681419221507 2023-04-13 20:53:43,411 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting af6566682db34fc88f98d68a9d721e11, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681419221676 2023-04-13 20:53:43,413 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ade28c1ba2084d3ab952d124de3abbe3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:43,448 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#9 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:43,450 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:43,493 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:43,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:43,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:43,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:43,504 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:43,504 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:43,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:43,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,508 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,508 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,509 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:43,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:43,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:43,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,512 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:43,512 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:43,512 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:43,514 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,514 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/e9a5792d2ade4751836e7e4f8c81aa4c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/e9a5792d2ade4751836e7e4f8c81aa4c 2023-04-13 20:53:43,514 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:43,515 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:53:43,515 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:43,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:43,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:43,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,538 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into e9a5792d2ade4751836e7e4f8c81aa4c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:43,538 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:43,538 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=13, startTime=1681419223388; duration=0sec 2023-04-13 20:53:43,539 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:43,539 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:43,542 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:43,542 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:43,542 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,542 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=34.8 K 2023-04-13 20:53:43,543 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a21d5e10fb044536b389c1ba26daa828, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1681419221507 2023-04-13 20:53:43,544 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 75c3022945be471697fef4c24035d6e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681419221676 2023-04-13 20:53:43,545 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting def0aea8f60d4b319daa37701c1ab872, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:43,562 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/b4e4ac3b1a544065a73f57080f86c13e 2023-04-13 20:53:43,565 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#12 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:43,669 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/6144193ddd754afdbcac030f024cd58f 2023-04-13 20:53:43,732 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/9c8a0ad4272547e4895e3c702d80fb0e 2023-04-13 20:53:43,746 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/b4e4ac3b1a544065a73f57080f86c13e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e 2023-04-13 20:53:43,758 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e, entries=150, sequenceid=66, filesize=11.6 K 2023-04-13 20:53:43,761 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/6144193ddd754afdbcac030f024cd58f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f 2023-04-13 20:53:43,771 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f, entries=150, sequenceid=66, filesize=11.6 K 2023-04-13 20:53:43,773 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/9c8a0ad4272547e4895e3c702d80fb0e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e 2023-04-13 20:53:43,804 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e, entries=150, sequenceid=66, filesize=11.6 K 2023-04-13 20:53:43,807 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for cdf6485ca98235662ec10d4d33684acb in 293ms, sequenceid=66, compaction requested=false 2023-04-13 20:53:43,807 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:43,807 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:43,807 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,810 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:43,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:43,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,813 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34dda23a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:43,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:43,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,827 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:43,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:43,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:43,830 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:43,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:43,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:43,841 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:43,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:43,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:43,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:43,842 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:43,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:43,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,845 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:43,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:43,854 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:43,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:43,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:43,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:43,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:43,861 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:43,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:43,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:43,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:43,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:43,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:43,917 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/0db3094ebbf743a6a4d0d2b6ff390dbb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0db3094ebbf743a6a4d0d2b6ff390dbb 2023-04-13 20:53:43,961 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:43,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:43,970 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 0db3094ebbf743a6a4d0d2b6ff390dbb(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:43,970 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:43,971 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=13, startTime=1681419223388; duration=0sec 2023-04-13 20:53:43,977 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:43,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/cf7471a81e7746fba02dad247e87168c 2023-04-13 20:53:43,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:43,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:43,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:43,991 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:43,991 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:43,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:43,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:43,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:43,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,998 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:43,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:43,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:44,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:44,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:44,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:44,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:44,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:44,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:44,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:44,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:44,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:44,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:44,027 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:44,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332f5477[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:44,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:44,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:44,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:44,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:44,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419283998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419284008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:44,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:44,039 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:44,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:44,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:44,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:44,040 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:44,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419284026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419284037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:44,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:44,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419223981, max timestamp=1681419224002Num uniques 859; 2023-04-13 20:53:44,062 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 20:53:44,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419223981, max timestamp=1681419224012Num uniques 859; 2023-04-13 20:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419223981, max timestamp=1681419224001Num uniques 859; 2023-04-13 20:53:44,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,067 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 20:53:44,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 20:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/abd00b0889b84faaa3754dec8e0480c9 2023-04-13 20:53:44,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,074 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/e95f458bad524ddb83a9f7b3f830da66 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e95f458bad524ddb83a9f7b3f830da66 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:44,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:44,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:44,077 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:44,077 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:44,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:44,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:44,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:44,077 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:44,110 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into e95f458bad524ddb83a9f7b3f830da66(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:44,111 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:44,111 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=13, startTime=1681419223388; duration=0sec 2023-04-13 20:53:44,111 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:44,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8d4a3a08bac441b7acf3465d0bd755e9 2023-04-13 20:53:44,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/cf7471a81e7746fba02dad247e87168c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c 2023-04-13 20:53:44,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c, entries=200, sequenceid=78, filesize=13.9 K 2023-04-13 20:53:44,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/abd00b0889b84faaa3754dec8e0480c9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9 2023-04-13 20:53:44,177 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:44,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:44,180 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9, entries=150, sequenceid=78, filesize=11.6 K 2023-04-13 20:53:44,182 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8d4a3a08bac441b7acf3465d0bd755e9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9 2023-04-13 20:53:44,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:44,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:44,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:44,193 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:44,193 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:44,194 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:44,195 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:44,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,206 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:44,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:44,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:44,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:44,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:44,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9, entries=150, sequenceid=78, filesize=11.6 K 2023-04-13 20:53:44,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:44,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:44,210 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:44,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:44,212 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:44,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.28 KB/129315 for cdf6485ca98235662ec10d4d33684acb in 322ms, sequenceid=78, compaction requested=true 2023-04-13 20:53:44,213 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:44,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:44,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:44,213 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:44,213 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:44,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,214 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:44,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:44,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:44,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:44,214 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=132.99 KB heapSize=382.71 KB 2023-04-13 20:53:44,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:44,215 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:44,215 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:44,215 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:44,215 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,216 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0db3094ebbf743a6a4d0d2b6ff390dbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=37.2 K 2023-04-13 20:53:44,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:44,222 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:44,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:53:44,223 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0db3094ebbf743a6a4d0d2b6ff390dbb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:44,224 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:44,224 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:44,224 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:53:44,224 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,224 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:44,224 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:53:44,224 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/e9a5792d2ade4751836e7e4f8c81aa4c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=34.9 K 2023-04-13 20:53:44,225 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b4e4ac3b1a544065a73f57080f86c13e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681419222674 2023-04-13 20:53:44,226 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e9a5792d2ade4751836e7e4f8c81aa4c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:44,227 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cf7471a81e7746fba02dad247e87168c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223883 2023-04-13 20:53:44,228 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6144193ddd754afdbcac030f024cd58f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681419222674 2023-04-13 20:53:44,229 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting abd00b0889b84faaa3754dec8e0480c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223887 2023-04-13 20:53:44,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419284230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419284230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419284242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419284244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419284247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,273 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#19 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:44,273 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#20 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:44,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419284352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419284352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,380 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/99bf12cf3a0f4c91967ad10876986d1c 2023-04-13 20:53:44,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419284356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419284357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419284356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,399 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/8e876720cc924977add2ed6a72e45dab as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8e876720cc924977add2ed6a72e45dab 2023-04-13 20:53:44,418 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 8e876720cc924977add2ed6a72e45dab(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:44,418 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:44,418 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=13, startTime=1681419224213; duration=0sec 2023-04-13 20:53:44,418 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:44,421 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:44,427 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:44,427 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:44,428 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,428 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e95f458bad524ddb83a9f7b3f830da66, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=34.9 K 2023-04-13 20:53:44,432 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e95f458bad524ddb83a9f7b3f830da66, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1681419222420 2023-04-13 20:53:44,440 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9c8a0ad4272547e4895e3c702d80fb0e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1681419222674 2023-04-13 20:53:44,440 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/969862800ab1489393bbf79248452c4a 2023-04-13 20:53:44,442 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8d4a3a08bac441b7acf3465d0bd755e9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223887 2023-04-13 20:53:44,474 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#23 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:44,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419284556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,582 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=48.80 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/93cc6a7330a943219576a298ffc50f99 2023-04-13 20:53:44,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419284581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419284592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:44,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419284593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419284596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/99bf12cf3a0f4c91967ad10876986d1c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c 2023-04-13 20:53:44,638 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c, entries=150, sequenceid=119, filesize=11.6 K 2023-04-13 20:53:44,641 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/969862800ab1489393bbf79248452c4a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a 2023-04-13 20:53:44,657 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a, entries=150, sequenceid=119, filesize=11.6 K 2023-04-13 20:53:44,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/93cc6a7330a943219576a298ffc50f99 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99 2023-04-13 20:53:44,673 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99, entries=150, sequenceid=119, filesize=11.6 K 2023-04-13 20:53:44,677 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~146.41 KB/149925, heapSize ~422.74 KB/432888, currentSize=33.54 KB/34350 for cdf6485ca98235662ec10d4d33684acb in 463ms, sequenceid=119, compaction requested=false 2023-04-13 20:53:44,678 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:44,678 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,678 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:44,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:44,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:44,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:44,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:44,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12461884[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:44,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:44,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,689 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:44,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:44,689 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:44,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:44,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:44,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:44,701 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:44,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:44,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:44,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:44,701 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:44,705 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,705 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,705 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:44,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:44,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:44,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:44,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:44,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:44,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:44,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:44,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:44,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:44,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:44,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,717 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:44,717 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:44,797 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/590a1c344b90463985cfa087c7601438 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/590a1c344b90463985cfa087c7601438 2023-04-13 20:53:44,809 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 590a1c344b90463985cfa087c7601438(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:44,809 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:44,809 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=13, startTime=1681419224213; duration=0sec 2023-04-13 20:53:44,810 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:44,816 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:44,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:44,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:44,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:44,826 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:44,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:44,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:44,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:44,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:44,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:44,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:44,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:44,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:44,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:44,833 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:44,833 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:44,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:44,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:44,834 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:44,834 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:44,834 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:44,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:44,897 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:44,909 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/9fb82fd8e91541a193d7e707a10b8137 2023-04-13 20:53:44,965 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/14ec556c6b7e4bb6b2841ae257f6091c 2023-04-13 20:53:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419224956, max timestamp=1681419224968Num uniques 859; 2023-04-13 20:53:44,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:44,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419224956, max timestamp=1681419224968Num uniques 859; 2023-04-13 20:53:44,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:44,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419224956, max timestamp=1681419224969Num uniques 859; 2023-04-13 20:53:44,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:45,004 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/5b80d2f09f4c409593a55c2c5a52dae6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/5b80d2f09f4c409593a55c2c5a52dae6 2023-04-13 20:53:45,014 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 5b80d2f09f4c409593a55c2c5a52dae6(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:45,014 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:45,014 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=13, startTime=1681419224214; duration=0sec 2023-04-13 20:53:45,014 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:45,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1681419224993, max timestamp=1681419225029Num uniques 1018; 2023-04-13 20:53:45,043 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=14558, suffix heap size=133840,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=318 2023-04-13 20:53:45,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419225007, max timestamp=1681419225028Num uniques 1018; 2023-04-13 20:53:45,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:53:45,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419225007, max timestamp=1681419225029Num uniques 1018; 2023-04-13 20:53:45,061 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:53:45,062 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8f07cadf0678450f97966d2ad772f34e 2023-04-13 20:53:45,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/9fb82fd8e91541a193d7e707a10b8137 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137 2023-04-13 20:53:45,083 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137, entries=150, sequenceid=129, filesize=11.6 K 2023-04-13 20:53:45,089 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/14ec556c6b7e4bb6b2841ae257f6091c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c 2023-04-13 20:53:45,099 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c, entries=150, sequenceid=129, filesize=11.6 K 2023-04-13 20:53:45,101 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8f07cadf0678450f97966d2ad772f34e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e 2023-04-13 20:53:45,121 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e, entries=150, sequenceid=129, filesize=11.6 K 2023-04-13 20:53:45,123 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=140.89 KB/144270 for cdf6485ca98235662ec10d4d33684acb in 289ms, sequenceid=129, compaction requested=true 2023-04-13 20:53:45,123 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:45,124 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:45,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:45,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12672, totalHeapSize=29048, min timestamp=1681419225065, max timestamp=1681419225121Num uniques 1027; 2023-04-13 20:53:45,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=12672, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-13 20:53:45,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681419225065, max timestamp=1681419225121Num uniques 1177; 2023-04-13 20:53:45,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=12682, suffix heap size=150376,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=277 2023-04-13 20:53:45,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1681419225065, max timestamp=1681419225121Num uniques 1027; 2023-04-13 20:53:45,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=12682, suffix heap size=134776,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=277 2023-04-13 20:53:45,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=46.96 KB heapSize=114.94 KB 2023-04-13 20:53:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:45,130 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:45,131 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:45,131 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:45,133 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:53:45,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:45,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b48b4f4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:45,139 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:45,139 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:45,141 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:45,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:45,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:45,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:45,186 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:45,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:45,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:45,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:45,186 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:45,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419225140, max timestamp=1681419225214Num uniques 859; 2023-04-13 20:53:45,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419225140, max timestamp=1681419225214Num uniques 859; 2023-04-13 20:53:45,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:53:45,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419225140, max timestamp=1681419225214Num uniques 859; 2023-04-13 20:53:45,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:53:45,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:53:45,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,254 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,254 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,254 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:45,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:45,254 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,255 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:45,255 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:45,305 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-13 20:53:45,307 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-13 20:53:45,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419285330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419285333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419285341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419285342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419285344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,355 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:45,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:45,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:45,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:45,371 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:45,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:45,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:45,374 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,374 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:45,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:45,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:45,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:45,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:45,379 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:45,389 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,389 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:45,389 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:45,389 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,391 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725d0fd5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:45,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:45,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,395 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:45,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:45,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:45,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:45,408 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:45,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:45,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:45,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:45,409 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:45,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:45,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:45,421 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:45,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419285448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419285448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419285453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419285453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,458 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419285453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,521 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:45,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:45,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:45,527 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:45,527 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:45,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:45,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,538 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:45,538 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:45,538 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:45,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:45,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:45,540 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:45,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:45,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:45,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:45,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:45,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b987105[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:45,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:45,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:45,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:45,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:45,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:45,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:45,558 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:45,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:45,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:45,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:45,558 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:45,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:45,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:45,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,574 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:45,574 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,575 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,575 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,575 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,618 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/89f973198a6141e69417530262154a12 2023-04-13 20:53:45,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419285653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419285654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419285659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419285660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419285660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,674 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:45,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:45,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:45,694 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:45,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:45,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:45,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:45,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,707 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:45,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:45,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:45,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:45,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b8dcca[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:45,715 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:45,715 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:45,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:45,726 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,726 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:45,727 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,727 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:45,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:45,755 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:45,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:45,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:45,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:45,755 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:45,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:45,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,773 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:45,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,873 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:45,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:45,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:45,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:45,882 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:45,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:45,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:45,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,887 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,888 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:45,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:45,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:45,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:45,894 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,894 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:45,894 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:45,894 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:45,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7739241d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:45,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:45,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,900 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:45,900 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:45,900 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:45,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:45,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:45,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:45,906 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:45,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:45,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:45,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:45,906 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:45,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:45,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:45,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:45,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:45,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:45,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:45,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:45,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:45,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:45,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:45,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:45,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:45,916 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:45,917 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:45,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419285957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419285962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419285963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419285963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:45,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:45,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419285965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,017 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:46,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:46,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:46,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:46,022 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:46,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:46,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:46,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,025 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:46,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:46,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:46,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:46,029 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,029 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:46,029 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:46,029 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:46,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:46,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:46,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:46,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d00ae88[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:46,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:46,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:46,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:46,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:46,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:46,045 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:46,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:46,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:46,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:46,045 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:46,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:46,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:46,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,065 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:46,065 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/fc64db17f27846a3a021b65b425d3f51 2023-04-13 20:53:46,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8965b5a537df4fafb6f36e71eb10db46 2023-04-13 20:53:46,140 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/89f973198a6141e69417530262154a12 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12 2023-04-13 20:53:46,147 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12, entries=150, sequenceid=186, filesize=11.8 K 2023-04-13 20:53:46,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/fc64db17f27846a3a021b65b425d3f51 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51 2023-04-13 20:53:46,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51, entries=150, sequenceid=186, filesize=11.8 K 2023-04-13 20:53:46,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/8965b5a537df4fafb6f36e71eb10db46 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46 2023-04-13 20:53:46,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:46,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:46,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:46,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:46,172 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:46,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:46,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:46,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:46,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:46,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:46,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46, entries=150, sequenceid=186, filesize=11.8 K 2023-04-13 20:53:46,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:46,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:46,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,187 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:46,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~114.89 KB/117648, currentSize=154.31 KB/158010 for cdf6485ca98235662ec10d4d33684acb in 1058ms, sequenceid=186, compaction requested=true 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:46,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:46,188 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:46,189 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:46,189 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:46,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:46,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,191 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,191 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:46,191 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,191 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8e876720cc924977add2ed6a72e45dab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=46.8 K 2023-04-13 20:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,193 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8e876720cc924977add2ed6a72e45dab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223887 2023-04-13 20:53:46,193 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:46,193 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:46,193 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,194 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/590a1c344b90463985cfa087c7601438, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=46.8 K 2023-04-13 20:53:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,195 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 969862800ab1489393bbf79248452c4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681419224169 2023-04-13 20:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,196 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 590a1c344b90463985cfa087c7601438, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223887 2023-04-13 20:53:46,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:46,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1568ea74[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:46,196 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:46,196 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,196 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 99bf12cf3a0f4c91967ad10876986d1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681419224169 2023-04-13 20:53:46,196 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 14ec556c6b7e4bb6b2841ae257f6091c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681419224225 2023-04-13 20:53:46,197 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9fb82fd8e91541a193d7e707a10b8137, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681419224225 2023-04-13 20:53:46,197 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:46,198 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fc64db17f27846a3a021b65b425d3f51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:46,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,205 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 89f973198a6141e69417530262154a12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:46,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:46,205 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:46,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:46,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:46,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:46,205 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:46,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,214 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,215 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,222 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:46,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:46,222 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,223 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:46,223 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:46,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:46,230 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,238 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,268 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/0be99b581d1041fc9e6608726d35b54b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0be99b581d1041fc9e6608726d35b54b 2023-04-13 20:53:46,281 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 0be99b581d1041fc9e6608726d35b54b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:46,281 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,282 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=12, startTime=1681419226189; duration=0sec 2023-04-13 20:53:46,282 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,282 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:46,287 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:46,289 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:46,290 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,291 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/5b80d2f09f4c409593a55c2c5a52dae6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=46.8 K 2023-04-13 20:53:46,291 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5b80d2f09f4c409593a55c2c5a52dae6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1681419223887 2023-04-13 20:53:46,292 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 93cc6a7330a943219576a298ffc50f99, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1681419224169 2023-04-13 20:53:46,294 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8f07cadf0678450f97966d2ad772f34e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1681419224225 2023-04-13 20:53:46,295 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8965b5a537df4fafb6f36e71eb10db46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,312 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#32 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,323 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:46,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:46,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:46,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:46,334 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:46,334 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:46,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:46,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,337 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:46,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:46,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:46,342 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,342 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:46,343 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-04-13 20:53:46,343 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:46,343 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:46,343 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:46,343 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:46,343 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:46,344 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:46,368 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/1816517f5fca4f63be572011fe3fdea1 2023-04-13 20:53:46,430 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/41a2e4d476454df7bea50b61e20477c1 2023-04-13 20:53:46,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:46,466 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:46,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419286474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419286475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419286475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419286483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419286484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,492 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/43392edb255847a4ae75ae53671a9bba 2023-04-13 20:53:46,502 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/1816517f5fca4f63be572011fe3fdea1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1 2023-04-13 20:53:46,516 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1, entries=150, sequenceid=223, filesize=11.8 K 2023-04-13 20:53:46,517 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/41a2e4d476454df7bea50b61e20477c1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1 2023-04-13 20:53:46,534 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1, entries=150, sequenceid=223, filesize=11.8 K 2023-04-13 20:53:46,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/43392edb255847a4ae75ae53671a9bba as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba 2023-04-13 20:53:46,549 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba, entries=150, sequenceid=223, filesize=11.8 K 2023-04-13 20:53:46,552 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for cdf6485ca98235662ec10d4d33684acb in 210ms, sequenceid=223, compaction requested=false 2023-04-13 20:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,552 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:46,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:46,555 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,555 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:46,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ae93a9f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:46,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:46,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:46,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:46,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:46,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:46,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:46,572 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:46,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:46,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:46,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:46,572 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:46,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:46,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:46,590 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:46,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:53:46,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:46,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:46,599 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,599 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:46,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:46,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/a0fd29fc1495414c8f6e9f185573a6b1 2023-04-13 20:53:46,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419286643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419286645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419286648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419286648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,666 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/0c707070429f4b0b93aea1b770e49e8e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0c707070429f4b0b93aea1b770e49e8e 2023-04-13 20:53:46,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/9e4cdf1908ac43aabeef73827680018e 2023-04-13 20:53:46,677 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 0c707070429f4b0b93aea1b770e49e8e(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:46,677 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,678 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=12, startTime=1681419226188; duration=0sec 2023-04-13 20:53:46,678 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:46,690 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:46,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:46,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:46,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:46,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:46,698 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:46,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:46,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:46,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:46,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:46,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:46,707 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,707 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:46,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:46,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:46,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:46,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:46,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744ef613[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:46,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:46,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:46,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:46,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:46,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,720 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/f018d7dd850940e593d4c048144ce90d 2023-04-13 20:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:46,726 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,726 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:46,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:46,731 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:46,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:46,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:46,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:46,731 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:46,734 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/7ade46047be84656a6658d488dc2dee6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7ade46047be84656a6658d488dc2dee6 2023-04-13 20:53:46,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:46,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:46,742 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/a0fd29fc1495414c8f6e9f185573a6b1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1 2023-04-13 20:53:46,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,744 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 7ade46047be84656a6658d488dc2dee6(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,745 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:46,746 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=12, startTime=1681419226189; duration=0sec 2023-04-13 20:53:46,746 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:46,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1, entries=150, sequenceid=235, filesize=11.8 K 2023-04-13 20:53:46,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/9e4cdf1908ac43aabeef73827680018e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e 2023-04-13 20:53:46,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:46,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:46,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,756 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:46,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:46,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,759 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,759 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:46,759 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,759 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:46,759 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:46,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419286754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419286754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419286760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419286760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e, entries=150, sequenceid=235, filesize=11.8 K 2023-04-13 20:53:46,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/f018d7dd850940e593d4c048144ce90d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d 2023-04-13 20:53:46,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d, entries=150, sequenceid=235, filesize=11.8 K 2023-04-13 20:53:46,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cdf6485ca98235662ec10d4d33684acb in 180ms, sequenceid=235, compaction requested=true 2023-04-13 20:53:46,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,776 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:46,777 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:46,777 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:46,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,778 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:46,778 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:46,778 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:46,778 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:46,779 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,779 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,779 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0be99b581d1041fc9e6608726d35b54b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.6 K 2023-04-13 20:53:46,779 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0c707070429f4b0b93aea1b770e49e8e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.6 K 2023-04-13 20:53:46,787 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0c707070429f4b0b93aea1b770e49e8e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,787 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0be99b581d1041fc9e6608726d35b54b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,789 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 41a2e4d476454df7bea50b61e20477c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681419225329 2023-04-13 20:53:46,789 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1816517f5fca4f63be572011fe3fdea1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681419225329 2023-04-13 20:53:46,789 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9e4cdf1908ac43aabeef73827680018e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:46,789 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a0fd29fc1495414c8f6e9f185573a6b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:46,811 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,812 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#40 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,824 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 20:53:46,824 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-13 20:53:46,856 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/5f9b40c67a8d4e2880e108fb9ec30533 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5f9b40c67a8d4e2880e108fb9ec30533 2023-04-13 20:53:46,857 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/006dec6be9e648a787c544be9673634d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/006dec6be9e648a787c544be9673634d 2023-04-13 20:53:46,859 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:46,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:46,868 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 006dec6be9e648a787c544be9673634d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:46,868 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,868 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=13, startTime=1681419226776; duration=0sec 2023-04-13 20:53:46,869 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:46,869 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:46,869 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 5f9b40c67a8d4e2880e108fb9ec30533(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:46,869 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:46,869 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=13, startTime=1681419226777; duration=0sec 2023-04-13 20:53:46,869 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:46,872 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:46,872 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:46,873 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,873 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7ade46047be84656a6658d488dc2dee6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.6 K 2023-04-13 20:53:46,873 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7ade46047be84656a6658d488dc2dee6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1681419225119 2023-04-13 20:53:46,874 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 43392edb255847a4ae75ae53671a9bba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1681419225329 2023-04-13 20:53:46,874 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f018d7dd850940e593d4c048144ce90d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:46,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:46,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:46,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:46,882 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:46,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:46,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:46,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,885 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:46,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:46,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:46,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:46,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:46,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:46,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:46,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:46,897 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:46,898 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:46,899 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:53:46,900 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:46,900 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:46,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:46,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:46,913 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#41 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:46,963 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/50bf302a3c2b4c2a89c1dc6675466b65 2023-04-13 20:53:46,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:46,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:47,019 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/eebb972a76d24868a6143e0ac873e118 2023-04-13 20:53:47,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419287034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419287035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419287036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419287041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419287143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419287149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419287157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419287160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,351 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/039b50fe2d12463e9e2f1ea9d372122d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/039b50fe2d12463e9e2f1ea9d372122d 2023-04-13 20:53:47,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419287347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419287354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,362 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 039b50fe2d12463e9e2f1ea9d372122d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:47,363 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:47,363 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=13, startTime=1681419226777; duration=0sec 2023-04-13 20:53:47,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,363 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:47,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419287362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419287363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419287495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,504 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/bdf2353039684be595121fa94d948bcc 2023-04-13 20:53:47,513 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/50bf302a3c2b4c2a89c1dc6675466b65 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65 2023-04-13 20:53:47,519 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65, entries=150, sequenceid=259, filesize=11.8 K 2023-04-13 20:53:47,521 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/eebb972a76d24868a6143e0ac873e118 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118 2023-04-13 20:53:47,531 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118, entries=150, sequenceid=259, filesize=11.8 K 2023-04-13 20:53:47,532 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/bdf2353039684be595121fa94d948bcc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc 2023-04-13 20:53:47,539 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc, entries=150, sequenceid=259, filesize=11.8 K 2023-04-13 20:53:47,540 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cdf6485ca98235662ec10d4d33684acb in 642ms, sequenceid=259, compaction requested=false 2023-04-13 20:53:47,540 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:47,540 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:47,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:47,541 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:47,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:47,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c5262fc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:47,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:47,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:47,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:47,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:47,550 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:47,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:47,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:47,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:47,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:47,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:47,554 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:47,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:47,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:47,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:47,578 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:47,578 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:47,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:47,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:47,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:47,678 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:47,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:47,688 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:47,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:47,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:47,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:47,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,691 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:47,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:47,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:47,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419287692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419287695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:47,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:47,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:47,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419287699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419287700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:47,716 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:47,716 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:47,716 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:47,716 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:47,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:47,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:47,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:47,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e3b1048[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:47,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:47,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,722 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:47,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:47,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:47,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:47,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:47,728 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:47,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:47,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:47,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:47,729 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:47,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:47,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,740 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:47,741 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:47,741 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:47,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419287802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419287802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419287805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419287805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,841 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:47,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:47,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:47,847 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:47,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:47,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:47,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:47,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:47,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:47,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:47,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:47,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:47,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:47,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:47,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:47,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:47,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:47,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:47,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:47,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d85e6d2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:47,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:47,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:47,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:47,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:47,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:47,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:47,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:47,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:47,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:47,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:47,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:47,908 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:47,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:47,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:47,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:47,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:47,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:47,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:47,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:47,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:47,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:47,925 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:47,925 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:47,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419288007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419288010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419288010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419288011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,025 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,050 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,057 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,057 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,057 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,058 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,058 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,058 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e786f1f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,064 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,064 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,065 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,065 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,065 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,070 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,070 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,070 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,070 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,070 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,080 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/f825118f778d4b33ac875577490dd27e 2023-04-13 20:53:48,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/d4b8bf18f2624f6585ee3d5458c33c90 2023-04-13 20:53:48,170 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=273 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/7de3e4ed0d3845bd80b23e26ad2e6b66 2023-04-13 20:53:48,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/f825118f778d4b33ac875577490dd27e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e 2023-04-13 20:53:48,181 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,187 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e, entries=150, sequenceid=273, filesize=11.9 K 2023-04-13 20:53:48,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,191 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/d4b8bf18f2624f6585ee3d5458c33c90 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90 2023-04-13 20:53:48,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90, entries=150, sequenceid=273, filesize=11.9 K 2023-04-13 20:53:48,200 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/7de3e4ed0d3845bd80b23e26ad2e6b66 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66 2023-04-13 20:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41b2b1ae[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,206 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,211 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,211 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,215 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66, entries=150, sequenceid=273, filesize=11.9 K 2023-04-13 20:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,218 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for cdf6485ca98235662ec10d4d33684acb in 560ms, sequenceid=273, compaction requested=true 2023-04-13 20:53:48,218 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:48,219 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:48,219 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:48,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:48,220 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:48,220 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:48,220 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,221 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/006dec6be9e648a787c544be9673634d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.8 K 2023-04-13 20:53:48,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,221 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 006dec6be9e648a787c544be9673634d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:48,222 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,222 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,223 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,223 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,223 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,224 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:48,224 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:48,224 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,224 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5f9b40c67a8d4e2880e108fb9ec30533, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.8 K 2023-04-13 20:53:48,225 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5f9b40c67a8d4e2880e108fb9ec30533, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:48,225 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 50bf302a3c2b4c2a89c1dc6675466b65, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681419226631 2023-04-13 20:53:48,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,228 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting eebb972a76d24868a6143e0ac873e118, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681419226631 2023-04-13 20:53:48,229 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d4b8bf18f2624f6585ee3d5458c33c90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:48,229 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f825118f778d4b33ac875577490dd27e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:48,244 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:48,245 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#49 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:48,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:48,315 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:48,323 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,330 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419288330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,337 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419288331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419288333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419288334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,345 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,345 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,345 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,345 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,347 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579d5080[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,360 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,360 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,362 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,362 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,362 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,373 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,373 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419288436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419288440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419288440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419288441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,473 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,479 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,484 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,484 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,484 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,484 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,487 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,487 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35cc8fd7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,489 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,489 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,491 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,492 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,492 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,493 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,496 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,496 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,498 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,500 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,504 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,504 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,504 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,604 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,609 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,611 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,613 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,613 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,613 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,613 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,613 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,617 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c11ee6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,619 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,620 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,621 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,622 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,626 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,626 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,633 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,633 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,636 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,637 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,637 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419288639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419288644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42364 deadline: 1681419288645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419288645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,671 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/5a17b8a8b9454ed2a82295c0d2aad108 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5a17b8a8b9454ed2a82295c0d2aad108 2023-04-13 20:53:48,679 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 5a17b8a8b9454ed2a82295c0d2aad108(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:48,679 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:48,680 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=13, startTime=1681419228219; duration=0sec 2023-04-13 20:53:48,680 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:48,680 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:53:48,682 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/4e8f98ef46c341fb93cea81fa7fe2090 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/4e8f98ef46c341fb93cea81fa7fe2090 2023-04-13 20:53:48,683 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:53:48,683 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:48,683 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,683 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/039b50fe2d12463e9e2f1ea9d372122d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=35.8 K 2023-04-13 20:53:48,684 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 039b50fe2d12463e9e2f1ea9d372122d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1681419226592 2023-04-13 20:53:48,685 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bdf2353039684be595121fa94d948bcc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1681419226631 2023-04-13 20:53:48,685 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7de3e4ed0d3845bd80b23e26ad2e6b66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:48,691 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 4e8f98ef46c341fb93cea81fa7fe2090(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:48,700 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:48,700 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=13, startTime=1681419228219; duration=0sec 2023-04-13 20:53:48,700 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:48,702 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:48,721 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/31437c8bd27e4a929a6a5c6f7a674093 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/31437c8bd27e4a929a6a5c6f7a674093 2023-04-13 20:53:48,731 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 31437c8bd27e4a929a6a5c6f7a674093(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:48,731 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:48,731 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=13, startTime=1681419228219; duration=0sec 2023-04-13 20:53:48,731 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:48,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/ebecb35588f54701bcad2534ba25a7ee 2023-04-13 20:53:48,737 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,747 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,751 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,751 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,780 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,780 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,780 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,780 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:48,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:48,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:48,784 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,786 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/c21868e8539a4d56b14dacdcdd6ba597 2023-04-13 20:53:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:48,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c2ae791[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:48,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:48,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:48,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:48,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:48,792 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:48,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:48,796 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:48,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:48,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:48,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:48,796 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:48,803 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,803 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,803 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:48,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:48,821 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:48,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:48,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:48,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:48,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:48,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:48,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:48,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:48,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:48,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:48,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/76180ae7a37d417da9c9233f5ad4de71 2023-04-13 20:53:48,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/ebecb35588f54701bcad2534ba25a7ee as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee 2023-04-13 20:53:48,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee, entries=200, sequenceid=294, filesize=14.3 K 2023-04-13 20:53:48,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/c21868e8539a4d56b14dacdcdd6ba597 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597 2023-04-13 20:53:48,856 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597, entries=150, sequenceid=294, filesize=11.9 K 2023-04-13 20:53:48,857 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/76180ae7a37d417da9c9233f5ad4de71 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71 2023-04-13 20:53:48,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71, entries=150, sequenceid=294, filesize=11.9 K 2023-04-13 20:53:48,863 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for cdf6485ca98235662ec10d4d33684acb in 548ms, sequenceid=294, compaction requested=false 2023-04-13 20:53:48,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:48,926 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:48,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:48,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:48,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:48,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:48,931 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:48,931 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,931 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:48,931 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:48,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:48,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:48,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:48,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:48,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:48,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:48,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:48,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:48,935 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:48,935 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:48,935 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:48,935 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:48,936 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:48,944 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/78d5224fa1d84eeb9bc1dd17b42d08c7 2023-04-13 20:53:48,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:48,947 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:48,962 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/b4eb3af7db0d4a42a62daf72041e2a4f 2023-04-13 20:53:48,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419288971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419288972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,978 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/951ac88e500b40e08e7678224a870bab 2023-04-13 20:53:48,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419288977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419228967, max timestamp=1681419228975Num uniques 859; 2023-04-13 20:53:48,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:48,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419228967, max timestamp=1681419228975Num uniques 859; 2023-04-13 20:53:48,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:48,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419228967, max timestamp=1681419228975Num uniques 859; 2023-04-13 20:53:48,983 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:53:48,984 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/78d5224fa1d84eeb9bc1dd17b42d08c7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7 2023-04-13 20:53:48,992 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7, entries=150, sequenceid=309, filesize=11.9 K 2023-04-13 20:53:48,994 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/b4eb3af7db0d4a42a62daf72041e2a4f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f 2023-04-13 20:53:49,006 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f, entries=150, sequenceid=309, filesize=11.9 K 2023-04-13 20:53:49,007 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/951ac88e500b40e08e7678224a870bab as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab 2023-04-13 20:53:49,023 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab, entries=150, sequenceid=309, filesize=11.9 K 2023-04-13 20:53:49,024 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 89ms, sequenceid=309, compaction requested=true 2023-04-13 20:53:49,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:49,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:49,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:49,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:49,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:49,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:49,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510eb3c4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:49,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:49,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:49,040 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=73.80 KB heapSize=205.69 KB 2023-04-13 20:53:49,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:49,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:49,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:49,041 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:49,041 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:53:49,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:49,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:49,049 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:49,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:49,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:49,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:49,050 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:49,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/eb4c24167d5641ba8664156eb95cc270 2023-04-13 20:53:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,057 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:49,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:49,061 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:49,061 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:49,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/4a1880fb0f714b28913f149b5851886e 2023-04-13 20:53:49,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419289090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419289091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419289097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419289097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,161 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:49,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:49,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:49,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:49,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:49,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:49,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:49,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:49,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:49,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:49,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:49,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:49,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:49,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:49,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419289199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419289205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419289206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419289206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:49,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed2c05[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:49,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:49,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,218 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:49,218 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:49,219 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:49,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:49,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,221 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:49,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:49,224 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:49,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:49,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:49,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:49,224 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:49,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,226 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,228 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:49,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:49,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:49,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:49,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:49,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:49,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:49,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:49,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:49,336 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:49,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:49,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:49,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,338 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,339 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,339 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:49,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:49,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:49,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:49,344 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:49,345 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:49,347 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:49,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:49,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30eb24d0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:49,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:49,351 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:49,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:49,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,352 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:49,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:49,356 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:49,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:49,356 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:49,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,359 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,362 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:49,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:49,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,367 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:49,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,367 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419289404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42354 deadline: 1681419289415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419289415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419289416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,467 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:49,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:49,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:49,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:49,476 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:49,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:49,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:49,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,478 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:49,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:49,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:49,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:49,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:49,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:49,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:49,490 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,490 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/639e5d91b1594761b79141700939255c 2023-04-13 20:53:49,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/eb4c24167d5641ba8664156eb95cc270 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270 2023-04-13 20:53:49,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:49,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b464b33[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:49,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:49,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,507 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:49,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:49,507 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,507 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:49,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270, entries=200, sequenceid=337, filesize=14.3 K 2023-04-13 20:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:49,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/4a1880fb0f714b28913f149b5851886e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e 2023-04-13 20:53:49,510 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419289508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:49,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,513 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:49,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:49,519 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:49,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:49,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:49,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:49,520 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:49,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e, entries=150, sequenceid=337, filesize=11.9 K 2023-04-13 20:53:49,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/639e5d91b1594761b79141700939255c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c 2023-04-13 20:53:49,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:49,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:49,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:49,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:49,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:49,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:49,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:49,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,530 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:49,530 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:49,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:49,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c, entries=150, sequenceid=337, filesize=11.9 K 2023-04-13 20:53:49,534 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.64 KB/210576, currentSize=114.05 KB/116790 for cdf6485ca98235662ec10d4d33684acb in 494ms, sequenceid=337, compaction requested=true 2023-04-13 20:53:49,534 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:49,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:49,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:49,534 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:49,534 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:49,534 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:49,537 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:49,537 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:49,537 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,537 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5a17b8a8b9454ed2a82295c0d2aad108, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=48.1 K 2023-04-13 20:53:49,538 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54170 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:49,538 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:49,538 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,538 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/4e8f98ef46c341fb93cea81fa7fe2090, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=52.9 K 2023-04-13 20:53:49,538 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5a17b8a8b9454ed2a82295c0d2aad108, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:49,540 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 4e8f98ef46c341fb93cea81fa7fe2090, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:49,540 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c21868e8539a4d56b14dacdcdd6ba597, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681419227693 2023-04-13 20:53:49,543 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ebecb35588f54701bcad2534ba25a7ee, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681419227687 2023-04-13 20:53:49,543 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b4eb3af7db0d4a42a62daf72041e2a4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419228329 2023-04-13 20:53:49,544 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 78d5224fa1d84eeb9bc1dd17b42d08c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419228329 2023-04-13 20:53:49,544 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4a1880fb0f714b28913f149b5851886e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:49,544 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting eb4c24167d5641ba8664156eb95cc270, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:49,570 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:49,571 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:49,597 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/2b4139ea619e44938f8cdd2f91508392 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/2b4139ea619e44938f8cdd2f91508392 2023-04-13 20:53:49,600 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/18133673a3b34eb99dcfc4d3e3d5fe83 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/18133673a3b34eb99dcfc4d3e3d5fe83 2023-04-13 20:53:49,610 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 18133673a3b34eb99dcfc4d3e3d5fe83(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:49,610 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:49,610 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=12, startTime=1681419229534; duration=0sec 2023-04-13 20:53:49,611 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:49,611 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:53:49,614 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:53:49,614 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:49,614 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 2b4139ea619e44938f8cdd2f91508392(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:49,614 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,614 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:49,614 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=12, startTime=1681419229534; duration=0sec 2023-04-13 20:53:49,614 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/31437c8bd27e4a929a6a5c6f7a674093, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=48.1 K 2023-04-13 20:53:49,614 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:49,615 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 31437c8bd27e4a929a6a5c6f7a674093, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=273, earliestPutTs=1681419226971 2023-04-13 20:53:49,615 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 76180ae7a37d417da9c9233f5ad4de71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1681419227693 2023-04-13 20:53:49,616 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 951ac88e500b40e08e7678224a870bab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419228329 2023-04-13 20:53:49,617 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 639e5d91b1594761b79141700939255c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:49,629 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#62 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:49,631 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:49,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:49,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:49,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:49,642 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:49,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:49,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:49,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:49,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:49,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:49,649 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:49,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:49,652 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/e33b6c87d2744eb49fc48c72ae9ec7da as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e33b6c87d2744eb49fc48c72ae9ec7da 2023-04-13 20:53:49,663 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into e33b6c87d2744eb49fc48c72ae9ec7da(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:49,663 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:49,663 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=12, startTime=1681419229534; duration=0sec 2023-04-13 20:53:49,663 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:49,667 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/069142e09f0d47d0b1c8abfcd995e65b 2023-04-13 20:53:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:49,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419289721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419289721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419289723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419289724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42416 deadline: 1681419289825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419289829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419289826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:49,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:49,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419289829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419290032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42394 deadline: 1681419290032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419290033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419290033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,081 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/d124c6ec7cc645029d028151f67d5280 2023-04-13 20:53:50,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419290336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419290337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419290337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419290337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,519 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/d7759296146d4979bc14838545d47192 2023-04-13 20:53:50,526 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/069142e09f0d47d0b1c8abfcd995e65b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b 2023-04-13 20:53:50,539 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b, entries=150, sequenceid=359, filesize=11.9 K 2023-04-13 20:53:50,540 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/d124c6ec7cc645029d028151f67d5280 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280 2023-04-13 20:53:50,548 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280, entries=150, sequenceid=359, filesize=11.9 K 2023-04-13 20:53:50,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/d7759296146d4979bc14838545d47192 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192 2023-04-13 20:53:50,554 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192, entries=150, sequenceid=359, filesize=11.9 K 2023-04-13 20:53:50,556 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for cdf6485ca98235662ec10d4d33684acb in 907ms, sequenceid=359, compaction requested=false 2023-04-13 20:53:50,556 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:50,556 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:50,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:50,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f72490c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:50,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:50,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:50,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:50,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:50,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:50,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:50,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:50,577 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:50,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:50,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:50,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:50,577 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:50,579 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,579 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:50,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:50,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:50,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:50,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:50,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:50,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:50,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,587 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:50,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:50,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:50,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:50,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:50,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:50,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:50,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:50,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:50,692 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:50,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:50,693 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:50,693 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:50,693 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,693 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:50,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,699 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:50,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:50,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:50,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:50,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:50,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:50,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:50,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:50,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:50,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:50,714 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:53:50,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:50,715 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:50,715 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:50,715 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:50,715 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:50,715 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:50,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:50,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:50,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419290860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,863 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419290861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419290862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419290863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419290965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419290967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419290968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:50,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419290969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,135 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/008af544e06d4bd4b8fd55661df0b15f 2023-04-13 20:53:51,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419291170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419291171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419291172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419291173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419291474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419291476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419291477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419291478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,559 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/6ce2bd20996a4ffca04e295281cf1e85 2023-04-13 20:53:51,583 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/9c8e08f7b1a8416bae6ed271173377b9 2023-04-13 20:53:51,591 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/008af544e06d4bd4b8fd55661df0b15f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f 2023-04-13 20:53:51,600 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f, entries=150, sequenceid=372, filesize=11.9 K 2023-04-13 20:53:51,602 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/6ce2bd20996a4ffca04e295281cf1e85 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85 2023-04-13 20:53:51,607 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85, entries=150, sequenceid=372, filesize=11.9 K 2023-04-13 20:53:51,609 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/9c8e08f7b1a8416bae6ed271173377b9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9 2023-04-13 20:53:51,613 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9, entries=150, sequenceid=372, filesize=11.9 K 2023-04-13 20:53:51,614 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for cdf6485ca98235662ec10d4d33684acb in 900ms, sequenceid=372, compaction requested=true 2023-04-13 20:53:51,615 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:51,615 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:51,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:51,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:51,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,618 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,618 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4616d5e3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:51,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:51,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,622 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:51,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:51,624 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:51,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:51,627 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:51,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:51,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:51,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:51,627 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:51,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:51,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:51,631 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,632 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:51,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:51,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:51,635 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:51,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:51,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:51,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:51,635 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,635 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,635 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:51,635 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,635 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:51,635 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:51,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:51,735 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:51,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:51,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:51,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:51,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:51,741 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:51,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:51,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:51,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:51,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:51,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:51,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:51,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:51,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:51,748 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:51,755 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:51,755 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:51,756 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:51,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:51,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:52,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,170 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/75dda27572bf4728b61c48b1678642c0 2023-04-13 20:53:52,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,584 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/a4780ac045c641af81cf75ac4efae411 2023-04-13 20:53:52,597 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/dc2b3d893fc3488f9425ad70d5145aac 2023-04-13 20:53:52,602 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/75dda27572bf4728b61c48b1678642c0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0 2023-04-13 20:53:52,606 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0, entries=150, sequenceid=392, filesize=11.9 K 2023-04-13 20:53:52,607 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/a4780ac045c641af81cf75ac4efae411 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411 2023-04-13 20:53:52,614 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411, entries=150, sequenceid=392, filesize=11.9 K 2023-04-13 20:53:52,615 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/dc2b3d893fc3488f9425ad70d5145aac as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac 2023-04-13 20:53:52,621 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac, entries=150, sequenceid=392, filesize=11.9 K 2023-04-13 20:53:52,622 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=73.80 KB/75570 for cdf6485ca98235662ec10d4d33684acb in 867ms, sequenceid=392, compaction requested=true 2023-04-13 20:53:52,622 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:52,622 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:52,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,624 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f32b455[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:52,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:52,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,628 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:52,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:52,634 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:52,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:52,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:52,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:52,634 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:52,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:53:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:52,639 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,639 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:52,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:52,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:52,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,643 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:52,643 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:52,643 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,643 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,743 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:52,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:52,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:52,748 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:52,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:52,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:52,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:52,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:52,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:52,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:52,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:52,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:52,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:52,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:52,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:52,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:52,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16146d53[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:52,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:52,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:52,767 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,767 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:52,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:52,768 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:52,772 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:52,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:52,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:52,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:52,772 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:52,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,777 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:52,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:52,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:52,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:52,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,785 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:52,785 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:52,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,886 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:52,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:52,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:52,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:52,889 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:52,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:52,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:52,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:52,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:52,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:52,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:52,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:52,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:52,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:52,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:52,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:52,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:52,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:52,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b136cb4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:52,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:52,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:52,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:52,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:52,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:52,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:52,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:52,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:52,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:52,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:52,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:52,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:52,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:52,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:52,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:52,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:52,915 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:52,915 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:52,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:52,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:52,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:52,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:52,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419292963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419292963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419292965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:52,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:52,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419292965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,015 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,019 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,027 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5a0df8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,040 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,040 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,050 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,050 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/b9fbda750932407397f1e806bb8b8639 2023-04-13 20:53:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:53,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/9e2b211bb78c45eaa135b56414145a63 2023-04-13 20:53:53,150 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,154 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f08ada[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,169 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,173 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,173 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,175 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,184 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,184 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419293268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419293268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419293268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419293270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,284 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,289 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54bb0d42[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,313 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,319 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,319 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,333 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,333 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,333 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,333 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,333 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,333 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,433 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,436 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,436 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,436 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,436 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,438 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64fdbdad[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,451 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,451 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,455 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,455 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,458 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,462 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,462 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:53,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/345617486f7b4fd39aed5cc97b3ad118 2023-04-13 20:53:53,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/b9fbda750932407397f1e806bb8b8639 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639 2023-04-13 20:53:53,500 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639, entries=200, sequenceid=407, filesize=14.3 K 2023-04-13 20:53:53,501 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/9e2b211bb78c45eaa135b56414145a63 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63 2023-04-13 20:53:53,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63, entries=150, sequenceid=407, filesize=11.9 K 2023-04-13 20:53:53,507 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/345617486f7b4fd39aed5cc97b3ad118 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118 2023-04-13 20:53:53,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118, entries=150, sequenceid=407, filesize=11.9 K 2023-04-13 20:53:53,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for cdf6485ca98235662ec10d4d33684acb in 875ms, sequenceid=407, compaction requested=true 2023-04-13 20:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:53,513 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:53:53,513 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:53,513 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:53,515 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64052 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:53:53,515 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:53,515 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:53:53,515 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,515 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:53,515 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/18133673a3b34eb99dcfc4d3e3d5fe83, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=62.6 K 2023-04-13 20:53:53,515 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,516 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/2b4139ea619e44938f8cdd2f91508392, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=60.2 K 2023-04-13 20:53:53,516 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 18133673a3b34eb99dcfc4d3e3d5fe83, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:53,516 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2b4139ea619e44938f8cdd2f91508392, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:53,516 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 069142e09f0d47d0b1c8abfcd995e65b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681419229081 2023-04-13 20:53:53,517 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d124c6ec7cc645029d028151f67d5280, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681419229081 2023-04-13 20:53:53,517 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 008af544e06d4bd4b8fd55661df0b15f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681419229719 2023-04-13 20:53:53,517 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6ce2bd20996a4ffca04e295281cf1e85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681419229719 2023-04-13 20:53:53,518 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a4780ac045c641af81cf75ac4efae411, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681419230857 2023-04-13 20:53:53,518 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 75dda27572bf4728b61c48b1678642c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681419230857 2023-04-13 20:53:53,518 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9e2b211bb78c45eaa135b56414145a63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:53,518 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b9fbda750932407397f1e806bb8b8639, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:53,532 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#75 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:53,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:53,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:53,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:53,541 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:53,562 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/f0b2505680434e4aa23ddd50cb4581ec as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f0b2505680434e4aa23ddd50cb4581ec 2023-04-13 20:53:53,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,572 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into f0b2505680434e4aa23ddd50cb4581ec(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:53,572 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:53,572 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=11, startTime=1681419233513; duration=0sec 2023-04-13 20:53:53,572 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:53,572 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:53:53,575 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:53:53,575 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:53,575 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,575 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e33b6c87d2744eb49fc48c72ae9ec7da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=60.2 K 2023-04-13 20:53:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,580 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e33b6c87d2744eb49fc48c72ae9ec7da, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1681419228974 2023-04-13 20:53:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,580 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,580 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d7759296146d4979bc14838545d47192, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1681419229081 2023-04-13 20:53:53,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,581 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9c8e08f7b1a8416bae6ed271173377b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1681419229719 2023-04-13 20:53:53,582 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting dc2b3d893fc3488f9425ad70d5145aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1681419230857 2023-04-13 20:53:53,582 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 345617486f7b4fd39aed5cc97b3ad118, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:53,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,588 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,588 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,588 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,588 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419293585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,594 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/fde2b37fd07c4fdb987a8f8c7e141771 2023-04-13 20:53:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f434907[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,597 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,598 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/85a8264e1d25499388a4e59253f66683 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/85a8264e1d25499388a4e59253f66683 2023-04-13 20:53:53,598 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,600 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,600 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,605 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into 85a8264e1d25499388a4e59253f66683(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:53,605 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:53,605 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=11, startTime=1681419233513; duration=0sec 2023-04-13 20:53:53,605 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,606 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,606 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,606 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,606 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,606 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,610 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,610 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,612 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,615 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,616 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,616 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:53,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/8bb3da5c4e694039865794d9f0e9f148 2023-04-13 20:53:53,646 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/3f3d337e902f4fb3bf50c196586acf5c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3f3d337e902f4fb3bf50c196586acf5c 2023-04-13 20:53:53,658 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 3f3d337e902f4fb3bf50c196586acf5c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:53,658 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:53,658 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=11, startTime=1681419233513; duration=0sec 2023-04-13 20:53:53,659 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:53,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419293695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,716 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,725 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,730 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,732 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,732 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,734 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,736 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,736 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,736 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,736 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a00a86f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,742 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,742 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,749 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,749 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,751 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,751 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,751 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,751 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:53,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,758 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,758 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,758 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419293774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419293774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419293775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419293778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,859 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:53,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:53,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:53,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:53,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:53,864 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:53,864 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,864 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:53,864 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:53,865 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,865 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,866 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,866 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,866 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,866 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:53,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:53,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:53,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:53,869 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,869 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:53,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:53,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:53,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:53,875 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:53,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65d6570[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:53,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:53,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:53,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:53,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:53,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:53,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:53,893 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:53,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:53,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:53,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:53,893 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:53,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:53,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:53,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419293898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:53,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:53,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:53,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:53,904 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:53,904 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:53,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:53,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:54,005 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:54,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:54,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:54,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:54,009 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:54,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,012 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:54,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:54,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:54,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:54,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:54,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:54,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:54,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:54,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:54,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:54,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:54,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:54,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:54,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:54,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:54,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:54,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:54,017 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:54,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:54,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:54,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc4b0c3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:54,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:54,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:54,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:54,027 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:54,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:54,032 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:54,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:54,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:54,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:54,032 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:54,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:54,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:54,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:54,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:54,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:54,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:54,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:54,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:54,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:54,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:54,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:54,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:54,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:54,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/03e29cda01cb46119c59def3c4d899cc 2023-04-13 20:53:54,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/fde2b37fd07c4fdb987a8f8c7e141771 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771 2023-04-13 20:53:54,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771, entries=200, sequenceid=426, filesize=14.3 K 2023-04-13 20:53:54,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/8bb3da5c4e694039865794d9f0e9f148 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148 2023-04-13 20:53:54,103 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148, entries=150, sequenceid=426, filesize=11.9 K 2023-04-13 20:53:54,104 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/03e29cda01cb46119c59def3c4d899cc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc 2023-04-13 20:53:54,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc, entries=150, sequenceid=426, filesize=11.9 K 2023-04-13 20:53:54,111 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 578ms, sequenceid=426, compaction requested=false 2023-04-13 20:53:54,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:54,144 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:54,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:54,147 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:54,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:54,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:54,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:54,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:54,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:54,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:54,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:54,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:54,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:54,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:54,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:54,152 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:54,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:54,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:54,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419294255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419294357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419294561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,577 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/e6846c2fcb954371ba1f24d5edc1695b 2023-04-13 20:53:54,590 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/048816b4c69e4944989b5359526c9239 2023-04-13 20:53:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419294789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419294790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419294793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419294794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:54,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419294865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,001 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/320a9e08fb564a6fb0ef3550d497571a 2023-04-13 20:53:55,007 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/e6846c2fcb954371ba1f24d5edc1695b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b 2023-04-13 20:53:55,015 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b, entries=150, sequenceid=442, filesize=11.9 K 2023-04-13 20:53:55,016 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/048816b4c69e4944989b5359526c9239 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239 2023-04-13 20:53:55,036 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239, entries=150, sequenceid=442, filesize=11.9 K 2023-04-13 20:53:55,037 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/320a9e08fb564a6fb0ef3550d497571a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a 2023-04-13 20:53:55,044 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a, entries=150, sequenceid=442, filesize=11.9 K 2023-04-13 20:53:55,045 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cdf6485ca98235662ec10d4d33684acb in 893ms, sequenceid=442, compaction requested=true 2023-04-13 20:53:55,046 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:55,046 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:55,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:55,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:55,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3acb4d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:55,056 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:55,056 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,061 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:55,062 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,062 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:55,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:55,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:55,067 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:55,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:55,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:55,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:55,067 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:55,069 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:55,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,075 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,075 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,075 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:55,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:55,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:55,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,076 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:55,076 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:55,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:55,176 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:55,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:55,188 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:55,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:55,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:55,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:55,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:55,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:55,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,193 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,193 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:55,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:55,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:55,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:55,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:55,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:55,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:55,196 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:53:55,196 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:55,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:55,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:55,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,247 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/adcb1b2dbf34483c8066221722417f4a 2023-04-13 20:53:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:55,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:55,400 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42342 deadline: 1681419295398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:55,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419295502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,699 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/3cbfdc8c958c4f7f94dc739533a49ef0 2023-04-13 20:53:55,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419295707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,729 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/23cab79b653a4899bcfc448aa0694c38 2023-04-13 20:53:55,741 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/adcb1b2dbf34483c8066221722417f4a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a 2023-04-13 20:53:55,749 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a, entries=150, sequenceid=461, filesize=11.9 K 2023-04-13 20:53:55,750 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/3cbfdc8c958c4f7f94dc739533a49ef0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0 2023-04-13 20:53:55,755 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0, entries=150, sequenceid=461, filesize=11.9 K 2023-04-13 20:53:55,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/23cab79b653a4899bcfc448aa0694c38 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38 2023-04-13 20:53:55,762 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38, entries=150, sequenceid=461, filesize=11.9 K 2023-04-13 20:53:55,764 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 567ms, sequenceid=461, compaction requested=true 2023-04-13 20:53:55,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:55,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@757647b5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:55,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:55,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:55,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:55,778 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:55,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:55,783 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:55,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:55,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:55,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:55,783 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:55,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:55,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:55,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:55,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:55,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:55,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:55,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:55,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:55,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:55,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,794 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:55,794 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:55,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:55,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:55,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:55,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:55,894 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:55,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:55,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:55,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:55,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:55,899 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:55,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:55,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:55,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:55,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:55,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:55,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:55,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:55,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:55,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:55,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:55,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:55,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:55,903 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:55,904 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:55,924 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92 2023-04-13 20:53:55,959 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/b149eb8f95cc45f5aa7b413d9dca479d 2023-04-13 20:53:56,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:56,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:56,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419296056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419296161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419296365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,373 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=474 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/39c9b11d840d42abbc2357721e22d75b 2023-04-13 20:53:56,379 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92 2023-04-13 20:53:56,384 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92, entries=150, sequenceid=474, filesize=11.9 K 2023-04-13 20:53:56,386 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/b149eb8f95cc45f5aa7b413d9dca479d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d 2023-04-13 20:53:56,395 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d, entries=150, sequenceid=474, filesize=11.9 K 2023-04-13 20:53:56,396 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/39c9b11d840d42abbc2357721e22d75b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b 2023-04-13 20:53:56,401 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b, entries=150, sequenceid=474, filesize=11.9 K 2023-04-13 20:53:56,403 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cdf6485ca98235662ec10d4d33684acb in 499ms, sequenceid=474, compaction requested=true 2023-04-13 20:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:56,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:56,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:56,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@186dca8e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:56,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:56,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:56,413 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:56,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:56,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:56,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:56,415 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:56,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:56,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:56,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:56,416 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:56,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:56,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:56,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:56,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:56,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:56,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:56,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:56,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:56,422 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:56,422 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:56,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:56,522 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:56,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:56,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:56,526 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:56,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:56,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:56,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:56,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:56,529 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:56,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:56,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:56,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:56,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:56,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:56,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:56,531 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:56,531 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:56,532 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:56,532 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:56,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:56,723 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419296717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419296797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,804 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:56,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419296802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419296804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,811 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:56,812 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4157 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:56,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419296814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,818 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:53:56,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419296825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:56,946 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/0dfb46ad61d74ddc85b628f3031847b9 2023-04-13 20:53:57,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419297030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:57,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419297334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,375 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cc48b491bef643cdabd37e339bd8119c 2023-04-13 20:53:57,413 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/f275c20ff65a4ff28f7ce9d9faf13410 2023-04-13 20:53:57,421 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/0dfb46ad61d74ddc85b628f3031847b9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9 2023-04-13 20:53:57,426 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9, entries=150, sequenceid=493, filesize=11.9 K 2023-04-13 20:53:57,427 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cc48b491bef643cdabd37e339bd8119c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c 2023-04-13 20:53:57,434 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c, entries=150, sequenceid=493, filesize=11.9 K 2023-04-13 20:53:57,436 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/f275c20ff65a4ff28f7ce9d9faf13410 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410 2023-04-13 20:53:57,442 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410, entries=150, sequenceid=493, filesize=11.9 K 2023-04-13 20:53:57,443 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 912ms, sequenceid=493, compaction requested=true 2023-04-13 20:53:57,443 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:57,443 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:57,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:57,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:57,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@320533e8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:57,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:57,448 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:57,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:57,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:57,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:57,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:57,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:57,461 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:57,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:57,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:57,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:57,461 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:57,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:57,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:57,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:57,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:57,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:57,469 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:57,469 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:57,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:57,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:57,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:57,569 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:57,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:57,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:57,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:57,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:57,576 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:57,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:57,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:57,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:57,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:57,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:57,578 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:57,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:57,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:57,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:57,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:57,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:57,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:57,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:57,584 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:57,584 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:57,584 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:53:57,584 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:57,584 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:57,584 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:57,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:57,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:57,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:57,594 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/fe327acc7c054510b3d4e0731ff3b91d 2023-04-13 20:53:57,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:57,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:57,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:57,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419297909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,015 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/df51591efeb24ca18a0354f0ff14b2bc 2023-04-13 20:53:58,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:58,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419298013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,041 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/6f1c9e6db12a41d597404557a0f5ca57 2023-04-13 20:53:58,049 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/fe327acc7c054510b3d4e0731ff3b91d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d 2023-04-13 20:53:58,054 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d, entries=150, sequenceid=506, filesize=11.9 K 2023-04-13 20:53:58,055 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/df51591efeb24ca18a0354f0ff14b2bc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc 2023-04-13 20:53:58,061 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc, entries=150, sequenceid=506, filesize=11.9 K 2023-04-13 20:53:58,062 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/6f1c9e6db12a41d597404557a0f5ca57 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57 2023-04-13 20:53:58,068 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57, entries=150, sequenceid=506, filesize=11.9 K 2023-04-13 20:53:58,072 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for cdf6485ca98235662ec10d4d33684acb in 488ms, sequenceid=506, compaction requested=true 2023-04-13 20:53:58,072 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:58,072 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:58,074 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,074 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,075 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:58,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e49e646[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:58,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:58,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,077 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:58,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:58,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:58,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:58,082 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:58,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:58,082 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:58,084 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,084 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,084 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,089 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:58,089 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,090 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,090 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,189 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:58,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:58,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:58,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:58,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:58,198 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:58,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:58,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:58,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,200 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:58,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:58,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:58,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,202 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:58,203 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,215 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/8038f91c8b9644aab396138e704ca45c 2023-04-13 20:53:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:58,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:58,227 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/0e015a502d2d4c27b1fcfceb13ec1ecb 2023-04-13 20:53:58,239 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/c8a14579c96b45f2be0cc1630eeb0a92 2023-04-13 20:53:58,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/8038f91c8b9644aab396138e704ca45c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c 2023-04-13 20:53:58,250 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c, entries=150, sequenceid=525, filesize=11.9 K 2023-04-13 20:53:58,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/0e015a502d2d4c27b1fcfceb13ec1ecb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb 2023-04-13 20:53:58,260 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb, entries=150, sequenceid=525, filesize=11.9 K 2023-04-13 20:53:58,261 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/c8a14579c96b45f2be0cc1630eeb0a92 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92 2023-04-13 20:53:58,267 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92, entries=150, sequenceid=525, filesize=11.9 K 2023-04-13 20:53:58,268 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 66ms, sequenceid=525, compaction requested=true 2023-04-13 20:53:58,268 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:58,268 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:58,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:58,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:58,273 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:58,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@594551d5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:58,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:58,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:58,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:58,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:58,280 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:58,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:58,292 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:58,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:58,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:58,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:58,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:58,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,295 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:58,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:58,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:58,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:58,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,310 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/870358ea07274f96b4170114b51b4f53 2023-04-13 20:53:58,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:58,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419298390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:58,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:58,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:58,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:58,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:58,427 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:58,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:58,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:58,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:58,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:58,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:58,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:58,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:58,452 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:58,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:58,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f8ac9c9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:58,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:58,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,458 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:58,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:58,458 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,458 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:58,463 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:58,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:58,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:58,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:58,463 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:58,465 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,465 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,465 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,467 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:58,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,477 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:58,477 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:58,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419298493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,577 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:58,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:58,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:58,581 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:58,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:58,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:58,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:58,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:58,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:58,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,587 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:58,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:58,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@348b25a3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:58,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:58,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:58,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:58,596 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:58,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:58,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:58,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:58,596 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:58,597 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,597 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,604 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:58,605 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,605 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:58,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:58,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419298697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,706 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:58,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:58,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:58,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:58,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:58,712 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:58,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:58,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:58,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:58,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:58,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:58,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:58,721 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,721 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:58,721 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:58,721 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:58,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,723 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:58,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65219ea2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:58,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:58,726 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,727 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:58,727 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:58,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:58,728 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:58,731 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:58,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:58,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:58,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:58,731 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:58,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:58,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:58,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,735 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:58,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:58,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:58,739 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,738 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:58,739 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:58,739 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:58,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:58,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:58,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/4e20d276f9d447569e9230cb10b04c1e 2023-04-13 20:53:58,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=539 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/15c1796ad56f4350869bc299ee693476 2023-04-13 20:53:58,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/870358ea07274f96b4170114b51b4f53 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53 2023-04-13 20:53:58,805 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53, entries=250, sequenceid=539, filesize=16.7 K 2023-04-13 20:53:58,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/4e20d276f9d447569e9230cb10b04c1e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e 2023-04-13 20:53:58,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e, entries=150, sequenceid=539, filesize=11.9 K 2023-04-13 20:53:58,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/15c1796ad56f4350869bc299ee693476 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476 2023-04-13 20:53:58,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476, entries=150, sequenceid=539, filesize=11.9 K 2023-04-13 20:53:58,817 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cdf6485ca98235662ec10d4d33684acb in 544ms, sequenceid=539, compaction requested=true 2023-04-13 20:53:58,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:58,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:58,819 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-13 20:53:58,819 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-13 20:53:58,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:58,819 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:58,823 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 117846 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-13 20:53:58,823 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:53:58,823 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,823 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/85a8264e1d25499388a4e59253f66683, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=115.1 K 2023-04-13 20:53:58,823 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110526 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-13 20:53:58,823 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 85a8264e1d25499388a4e59253f66683, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:58,823 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:53:58,824 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,824 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f0b2505680434e4aa23ddd50cb4581ec, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=107.9 K 2023-04-13 20:53:58,824 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fde2b37fd07c4fdb987a8f8c7e141771, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681419232654 2023-04-13 20:53:58,824 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f0b2505680434e4aa23ddd50cb4581ec, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:58,824 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e6846c2fcb954371ba1f24d5edc1695b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681419233564 2023-04-13 20:53:58,825 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8bb3da5c4e694039865794d9f0e9f148, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681419232654 2023-04-13 20:53:58,825 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting adcb1b2dbf34483c8066221722417f4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681419234230 2023-04-13 20:53:58,830 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 048816b4c69e4944989b5359526c9239, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681419233564 2023-04-13 20:53:58,830 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2bb9566dc8ee4fa28aeb3fe77f3c1a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681419235378 2023-04-13 20:53:58,830 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3cbfdc8c958c4f7f94dc739533a49ef0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681419234230 2023-04-13 20:53:58,830 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0dfb46ad61d74ddc85b628f3031847b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681419236045 2023-04-13 20:53:58,830 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b149eb8f95cc45f5aa7b413d9dca479d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681419235378 2023-04-13 20:53:58,831 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fe327acc7c054510b3d4e0731ff3b91d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681419236691 2023-04-13 20:53:58,831 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cc48b491bef643cdabd37e339bd8119c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681419236045 2023-04-13 20:53:58,831 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 8038f91c8b9644aab396138e704ca45c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681419237898 2023-04-13 20:53:58,831 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting df51591efeb24ca18a0354f0ff14b2bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681419236691 2023-04-13 20:53:58,831 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 870358ea07274f96b4170114b51b4f53, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238231 2023-04-13 20:53:58,832 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0e015a502d2d4c27b1fcfceb13ec1ecb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681419237898 2023-04-13 20:53:58,832 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4e20d276f9d447569e9230cb10b04c1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238248 2023-04-13 20:53:58,839 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:58,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:58,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:58,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:58,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:58,844 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:58,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:58,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:58,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:58,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:58,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:58,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:58,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:58,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:58,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:58,847 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:58,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:58,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:58,848 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:58,848 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,849 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:58,849 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:58,866 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#103 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:58,867 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#104 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:59,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:53:59,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:53:59,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419299069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:59,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419299178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,273 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/30713c0f043b406eb928703737509c69 2023-04-13 20:53:59,291 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/4f290a8255064f0f9eeb767ce95677c2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4f290a8255064f0f9eeb767ce95677c2 2023-04-13 20:53:59,292 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/e247f26c06ea41d597f96d3e8f731314 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e247f26c06ea41d597f96d3e8f731314 2023-04-13 20:53:59,292 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cb32c6dddb56469b92e6615979eb8011 2023-04-13 20:53:59,299 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 4f290a8255064f0f9eeb767ce95677c2(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:59,299 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:59,299 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=7, startTime=1681419238819; duration=0sec 2023-04-13 20:53:59,299 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:53:59,299 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 9 store files, 0 compacting, 9 eligible, 16 blocking 2023-04-13 20:53:59,300 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into e247f26c06ea41d597f96d3e8f731314(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:59,300 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:59,300 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=7, startTime=1681419238819; duration=0sec 2023-04-13 20:53:59,300 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:59,304 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 9 files of size 110526 starting at candidate #0 after considering 28 permutations with 28 in ratio 2023-04-13 20:53:59,304 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:53:59,304 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:59,304 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3f3d337e902f4fb3bf50c196586acf5c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=107.9 K 2023-04-13 20:53:59,305 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3f3d337e902f4fb3bf50c196586acf5c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1681419232015 2023-04-13 20:53:59,305 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 03e29cda01cb46119c59def3c4d899cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1681419232654 2023-04-13 20:53:59,306 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 320a9e08fb564a6fb0ef3550d497571a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1681419233564 2023-04-13 20:53:59,306 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 23cab79b653a4899bcfc448aa0694c38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1681419234230 2023-04-13 20:53:59,307 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 39c9b11d840d42abbc2357721e22d75b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=474, earliestPutTs=1681419235378 2023-04-13 20:53:59,307 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f275c20ff65a4ff28f7ce9d9faf13410, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1681419236045 2023-04-13 20:53:59,308 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6f1c9e6db12a41d597404557a0f5ca57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1681419236691 2023-04-13 20:53:59,308 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c8a14579c96b45f2be0cc1630eeb0a92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1681419237898 2023-04-13 20:53:59,309 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 15c1796ad56f4350869bc299ee693476, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238248 2023-04-13 20:53:59,334 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#107 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:53:59,352 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/3b86d80c66634bce9ea518ac638f328a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3b86d80c66634bce9ea518ac638f328a 2023-04-13 20:53:59,363 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 9 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 3b86d80c66634bce9ea518ac638f328a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:53:59,364 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:59,364 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=7, startTime=1681419238819; duration=0sec 2023-04-13 20:53:59,364 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:53:59,391 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:59,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419299389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:53:59,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419299692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,707 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/1abd6e1bb9f34235b6c871ee08d576fe 2023-04-13 20:53:59,722 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/30713c0f043b406eb928703737509c69 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69 2023-04-13 20:53:59,730 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69, entries=150, sequenceid=557, filesize=11.9 K 2023-04-13 20:53:59,731 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/cb32c6dddb56469b92e6615979eb8011 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011 2023-04-13 20:53:59,738 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011, entries=150, sequenceid=557, filesize=11.9 K 2023-04-13 20:53:59,739 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/1abd6e1bb9f34235b6c871ee08d576fe as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe 2023-04-13 20:53:59,746 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe, entries=150, sequenceid=557, filesize=11.9 K 2023-04-13 20:53:59,747 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for cdf6485ca98235662ec10d4d33684acb in 898ms, sequenceid=557, compaction requested=false 2023-04-13 20:53:59,747 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:53:59,747 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:53:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:53:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:53:59,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:53:59,749 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:59,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1619a02a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:53:59,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:53:59,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:53:59,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:53:59,756 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:53:59,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:53:59,759 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:53:59,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:53:59,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:53:59,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:53:59,759 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:53:59,761 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,761 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,761 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:53:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:53:59,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:59,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,763 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:59,769 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:59,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:53:59,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:59,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:53:59,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:53:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:53:59,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:53:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:53:59,770 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:53:59,770 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:53:59,870 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:53:59,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:53:59,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:53:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:53:59,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:53:59,873 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:53:59,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:59,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:53:59,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:53:59,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:59,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:53:59,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:53:59,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:53:59,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:53:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:59,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:53:59,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:53:59,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:53:59,879 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:53:59,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:00,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:00,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:00,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419300229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,318 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/1254d27e8a534c01a27b347b78b69285 2023-04-13 20:54:00,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419300332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,334 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/7c990624829f4f6d8a5d84d8ddb8aa95 2023-04-13 20:54:00,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419300535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,750 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/300216b435694d75a72df3228fd24f1f 2023-04-13 20:54:00,756 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/1254d27e8a534c01a27b347b78b69285 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285 2023-04-13 20:54:00,761 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285, entries=150, sequenceid=574, filesize=11.9 K 2023-04-13 20:54:00,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/7c990624829f4f6d8a5d84d8ddb8aa95 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95 2023-04-13 20:54:00,767 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95, entries=150, sequenceid=574, filesize=11.9 K 2023-04-13 20:54:00,768 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/300216b435694d75a72df3228fd24f1f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f 2023-04-13 20:54:00,772 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f, entries=150, sequenceid=574, filesize=11.9 K 2023-04-13 20:54:00,773 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for cdf6485ca98235662ec10d4d33684acb in 894ms, sequenceid=574, compaction requested=true 2023-04-13 20:54:00,773 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:00,773 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:00,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:00,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:00,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa4f7a1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:00,777 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:00,777 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:00,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:00,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:00,783 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:00,783 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:00,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:00,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:00,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:00,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:00,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,790 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:00,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:00,790 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:00,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:00,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:00,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:54:00,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:00,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419300839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419300839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419300840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42364 deadline: 1681419300846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42394 deadline: 1681419300846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,851 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:00,852 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8197 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:00,890 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:00,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:00,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:00,895 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:00,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:00,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:00,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:00,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,896 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,897 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:00,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:00,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:00,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:00,899 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:00,899 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:54:00,899 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:00,899 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:00,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:00,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ef71cb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:00,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:00,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:00,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:00,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:00,910 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:00,910 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:00,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:00,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:00,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:00,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:00,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:00,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:00,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:00,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:00,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:00,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:00,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:00,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:00,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:00,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:00,918 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:00,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:00,918 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:00,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:00,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419300942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419300943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:00,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:00,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419300943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,018 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:01,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:01,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:01,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:01,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:01,022 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:01,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:01,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:01,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,024 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:01,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:01,026 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,026 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:54:01,026 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:01,026 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:01,028 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:01,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b522bba[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:01,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:01,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:01,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:01,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:01,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:01,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:01,036 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:01,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:01,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:01,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:01,037 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:01,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,042 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:01,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:01,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:01,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:01,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:01,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419301145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:01,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:01,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419301149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:01,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419301149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:01,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:01,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:01,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:01,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:01,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:01,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:01,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:54:01,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:01,177 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:01,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:01,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3011ce0a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:01,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:01,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:01,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:01,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,185 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:01,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:01,188 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:01,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:01,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:01,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:01,188 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:01,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,191 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,195 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:01,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:01,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:01,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:01,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/2371b5668e9c4ba7a400d5620b3b60d0 2023-04-13 20:54:01,296 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:01,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:01,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:01,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:01,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:01,299 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:01,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:01,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:01,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:01,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:01,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:01,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:01,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:54:01,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:01,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:01,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:01,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c02e56[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:01,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:01,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,308 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:01,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:01,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:01,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:01,316 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:01,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:01,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:01,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:01,316 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:01,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:01,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:01,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:01,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,323 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:01,323 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:01,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,423 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:01,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:01,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:01,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:01,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:01,427 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:01,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:01,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:01,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:01,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:01,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. started... 2023-04-13 20:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. as already flushing 2023-04-13 20:54:01,435 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,438 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d879ac[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:01,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:01,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:01,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:01,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:01,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:01,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:01,447 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:01,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:01,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:01,448 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:01,448 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:01,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,451 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:54584 2023-04-13 20:54:01,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419301451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,451 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:01,451 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cd3abbd to 127.0.0.1:54584 2023-04-13 20:54:01,452 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:01,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,452 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4808c6d5 to 127.0.0.1:54584 2023-04-13 20:54:01,452 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:01,453 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:01,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:01,454 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:54584 2023-04-13 20:54:01,454 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:01,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419301454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419301455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:01,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:01,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:01,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:01,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:01,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:01,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:01,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:01,461 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:01,461 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:01,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:01,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:01,561 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:01,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:01,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/791926c773574742b87890c0363dd7fb 2023-04-13 20:54:01,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42342 deadline: 1681419301954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42416 deadline: 1681419301956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:01,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:01,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42354 deadline: 1681419301959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:02,055 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=593 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/59013771fdef414a9794bcd9450c052c 2023-04-13 20:54:02,062 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/2371b5668e9c4ba7a400d5620b3b60d0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0 2023-04-13 20:54:02,068 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0, entries=200, sequenceid=593, filesize=14.3 K 2023-04-13 20:54:02,069 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/791926c773574742b87890c0363dd7fb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb 2023-04-13 20:54:02,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb, entries=150, sequenceid=593, filesize=11.9 K 2023-04-13 20:54:02,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/59013771fdef414a9794bcd9450c052c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c 2023-04-13 20:54:02,090 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c, entries=150, sequenceid=593, filesize=11.9 K 2023-04-13 20:54:02,091 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for cdf6485ca98235662ec10d4d33684acb in 1269ms, sequenceid=593, compaction requested=true 2023-04-13 20:54:02,091 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:02,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:02,091 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:02,091 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:02,091 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:02,092 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:02,093 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52342 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:02,093 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/A is initiating minor compaction (all files) 2023-04-13 20:54:02,093 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:02,093 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/A in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:02,093 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/B is initiating minor compaction (all files) 2023-04-13 20:54:02,094 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e247f26c06ea41d597f96d3e8f731314, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=51.1 K 2023-04-13 20:54:02,094 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/B in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:02,094 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4f290a8255064f0f9eeb767ce95677c2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=48.7 K 2023-04-13 20:54:02,094 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4f290a8255064f0f9eeb767ce95677c2, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238248 2023-04-13 20:54:02,094 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e247f26c06ea41d597f96d3e8f731314, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238248 2023-04-13 20:54:02,095 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cb32c6dddb56469b92e6615979eb8011, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681419238366 2023-04-13 20:54:02,095 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 30713c0f043b406eb928703737509c69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681419238366 2023-04-13 20:54:02,095 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7c990624829f4f6d8a5d84d8ddb8aa95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1681419239058 2023-04-13 20:54:02,095 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1254d27e8a534c01a27b347b78b69285, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1681419239058 2023-04-13 20:54:02,096 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 791926c773574742b87890c0363dd7fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1681419240226 2023-04-13 20:54:02,096 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2371b5668e9c4ba7a400d5620b3b60d0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1681419240209 2023-04-13 20:54:02,126 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#A#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:02,128 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#B#compaction#115 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:02,551 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/f7f85f668efd423d86d064cd1579bb07 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f7f85f668efd423d86d064cd1579bb07 2023-04-13 20:54:02,558 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/3c7f1a6477774722945bd497daa9c017 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3c7f1a6477774722945bd497daa9c017 2023-04-13 20:54:02,561 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/A of cdf6485ca98235662ec10d4d33684acb into f7f85f668efd423d86d064cd1579bb07(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:02,561 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:02,561 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/A, priority=12, startTime=1681419242091; duration=0sec 2023-04-13 20:54:02,561 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:02,561 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:02,564 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:02,564 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): cdf6485ca98235662ec10d4d33684acb/C is initiating minor compaction (all files) 2023-04-13 20:54:02,564 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cdf6485ca98235662ec10d4d33684acb/C in TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:02,564 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3b86d80c66634bce9ea518ac638f328a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp, totalSize=48.7 K 2023-04-13 20:54:02,565 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 3b86d80c66634bce9ea518ac638f328a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=539, earliestPutTs=1681419238248 2023-04-13 20:54:02,565 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/B of cdf6485ca98235662ec10d4d33684acb into 3c7f1a6477774722945bd497daa9c017(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:02,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:02,565 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/B, priority=12, startTime=1681419242091; duration=0sec 2023-04-13 20:54:02,565 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1abd6e1bb9f34235b6c871ee08d576fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1681419238366 2023-04-13 20:54:02,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:02,566 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 300216b435694d75a72df3228fd24f1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=574, earliestPutTs=1681419239058 2023-04-13 20:54:02,566 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 59013771fdef414a9794bcd9450c052c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=593, earliestPutTs=1681419240226 2023-04-13 20:54:02,576 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): cdf6485ca98235662ec10d4d33684acb#C#compaction#116 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:02,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:02,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:54:02,959 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63859c62 to 127.0.0.1:54584 2023-04-13 20:54:02,959 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:02,959 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fea27db to 127.0.0.1:54584 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:54:02,960 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:54:02,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:02,965 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:54584 2023-04-13 20:54:02,965 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:02,994 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/40e3a67d39904bdcb15ea2df436ae8a8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/40e3a67d39904bdcb15ea2df436ae8a8 2023-04-13 20:54:02,999 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in cdf6485ca98235662ec10d4d33684acb/C of cdf6485ca98235662ec10d4d33684acb into 40e3a67d39904bdcb15ea2df436ae8a8(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:02,999 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:02,999 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb., storeName=cdf6485ca98235662ec10d4d33684acb/C, priority=12, startTime=1681419242092; duration=0sec 2023-04-13 20:54:02,999 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:03,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/a6fb981cc87a4fd38b791a40033e34f9 2023-04-13 20:54:03,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/5615a652f5b147babc16d1ed9cc72c04 2023-04-13 20:54:04,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/cd45548644894c20adcdee9294dd7f14 2023-04-13 20:54:04,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/a6fb981cc87a4fd38b791a40033e34f9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a6fb981cc87a4fd38b791a40033e34f9 2023-04-13 20:54:04,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a6fb981cc87a4fd38b791a40033e34f9, entries=150, sequenceid=610, filesize=11.9 K 2023-04-13 20:54:04,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/5615a652f5b147babc16d1ed9cc72c04 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5615a652f5b147babc16d1ed9cc72c04 2023-04-13 20:54:04,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5615a652f5b147babc16d1ed9cc72c04, entries=150, sequenceid=610, filesize=11.9 K 2023-04-13 20:54:04,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/cd45548644894c20adcdee9294dd7f14 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/cd45548644894c20adcdee9294dd7f14 2023-04-13 20:54:04,211 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/cd45548644894c20adcdee9294dd7f14, entries=150, sequenceid=610, filesize=11.9 K 2023-04-13 20:54:04,212 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=6.71 KB/6870 for cdf6485ca98235662ec10d4d33684acb in 1253ms, sequenceid=610, compaction requested=false 2023-04-13 20:54:04,212 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:10,911 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:54584 2023-04-13 20:54:10,912 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:10,917 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42927198 to 127.0.0.1:54584 2023-04-13 20:54:10,917 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 84 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 169 2023-04-13 20:54:10,917 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 68 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4094 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 3975 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1785 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5355 rows 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1769 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5297 rows 2023-04-13 20:54:10,918 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:54:10,918 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35594b1b to 127.0.0.1:54584 2023-04-13 20:54:10,918 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:10,922 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:54:10,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:54:10,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:10,937 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419250937"}]},"ts":"1681419250937"} 2023-04-13 20:54:10,938 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:54:10,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 20:54:10,940 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:54:10,941 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, UNASSIGN}] 2023-04-13 20:54:10,942 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, UNASSIGN 2023-04-13 20:54:10,943 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cdf6485ca98235662ec10d4d33684acb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:10,943 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419250943"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419250943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419250943"}]},"ts":"1681419250943"} 2023-04-13 20:54:10,944 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:11,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 20:54:11,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:11,102 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cdf6485ca98235662ec10d4d33684acb, disabling compactions & flushes 2023-04-13 20:54:11,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. after waiting 0 ms 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:11,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cdf6485ca98235662ec10d4d33684acb 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=A 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=B 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK cdf6485ca98235662ec10d4d33684acb, store=C 2023-04-13 20:54:11,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:11,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/8553bc46008b4cf0bc27567c0398618a 2023-04-13 20:54:11,132 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/ccd39f9a00d946c2ab5354f8125f96dd 2023-04-13 20:54:11,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=617 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/6ec86191c77b4b299f512f365510b459 2023-04-13 20:54:11,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/A/8553bc46008b4cf0bc27567c0398618a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8553bc46008b4cf0bc27567c0398618a 2023-04-13 20:54:11,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8553bc46008b4cf0bc27567c0398618a, entries=100, sequenceid=617, filesize=9.5 K 2023-04-13 20:54:11,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/B/ccd39f9a00d946c2ab5354f8125f96dd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/ccd39f9a00d946c2ab5354f8125f96dd 2023-04-13 20:54:11,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/ccd39f9a00d946c2ab5354f8125f96dd, entries=100, sequenceid=617, filesize=9.5 K 2023-04-13 20:54:11,174 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/.tmp/C/6ec86191c77b4b299f512f365510b459 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6ec86191c77b4b299f512f365510b459 2023-04-13 20:54:11,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6ec86191c77b4b299f512f365510b459, entries=100, sequenceid=617, filesize=9.5 K 2023-04-13 20:54:11,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for cdf6485ca98235662ec10d4d33684acb in 79ms, sequenceid=617, compaction requested=true 2023-04-13 20:54:11,211 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0db3094ebbf743a6a4d0d2b6ff390dbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/590a1c344b90463985cfa087c7601438, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0c707070429f4b0b93aea1b770e49e8e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/006dec6be9e648a787c544be9673634d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/4e8f98ef46c341fb93cea81fa7fe2090, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/18133673a3b34eb99dcfc4d3e3d5fe83, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/85a8264e1d25499388a4e59253f66683, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e247f26c06ea41d597f96d3e8f731314, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0] to archive 2023-04-13 20:54:11,213 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:11,219 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/49c1e436865b4c4eb508c1ead8a24488 2023-04-13 20:54:11,221 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/af6566682db34fc88f98d68a9d721e11 2023-04-13 20:54:11,223 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0db3094ebbf743a6a4d0d2b6ff390dbb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0db3094ebbf743a6a4d0d2b6ff390dbb 2023-04-13 20:54:11,226 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ade28c1ba2084d3ab952d124de3abbe3 2023-04-13 20:54:11,228 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b4e4ac3b1a544065a73f57080f86c13e 2023-04-13 20:54:11,231 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/cf7471a81e7746fba02dad247e87168c 2023-04-13 20:54:11,233 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/590a1c344b90463985cfa087c7601438 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/590a1c344b90463985cfa087c7601438 2023-04-13 20:54:11,234 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/99bf12cf3a0f4c91967ad10876986d1c 2023-04-13 20:54:11,236 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/9fb82fd8e91541a193d7e707a10b8137 2023-04-13 20:54:11,239 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0c707070429f4b0b93aea1b770e49e8e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0c707070429f4b0b93aea1b770e49e8e 2023-04-13 20:54:11,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 20:54:11,244 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/89f973198a6141e69417530262154a12 2023-04-13 20:54:11,246 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1816517f5fca4f63be572011fe3fdea1 2023-04-13 20:54:11,248 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/006dec6be9e648a787c544be9673634d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/006dec6be9e648a787c544be9673634d 2023-04-13 20:54:11,249 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a0fd29fc1495414c8f6e9f185573a6b1 2023-04-13 20:54:11,251 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/50bf302a3c2b4c2a89c1dc6675466b65 2023-04-13 20:54:11,253 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/4e8f98ef46c341fb93cea81fa7fe2090 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/4e8f98ef46c341fb93cea81fa7fe2090 2023-04-13 20:54:11,254 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f825118f778d4b33ac875577490dd27e 2023-04-13 20:54:11,256 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/ebecb35588f54701bcad2534ba25a7ee 2023-04-13 20:54:11,257 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/78d5224fa1d84eeb9bc1dd17b42d08c7 2023-04-13 20:54:11,259 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/eb4c24167d5641ba8664156eb95cc270 2023-04-13 20:54:11,261 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/18133673a3b34eb99dcfc4d3e3d5fe83 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/18133673a3b34eb99dcfc4d3e3d5fe83 2023-04-13 20:54:11,262 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/069142e09f0d47d0b1c8abfcd995e65b 2023-04-13 20:54:11,263 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/008af544e06d4bd4b8fd55661df0b15f 2023-04-13 20:54:11,264 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/75dda27572bf4728b61c48b1678642c0 2023-04-13 20:54:11,266 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/b9fbda750932407397f1e806bb8b8639 2023-04-13 20:54:11,268 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/85a8264e1d25499388a4e59253f66683 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/85a8264e1d25499388a4e59253f66683 2023-04-13 20:54:11,270 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fde2b37fd07c4fdb987a8f8c7e141771 2023-04-13 20:54:11,271 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e6846c2fcb954371ba1f24d5edc1695b 2023-04-13 20:54:11,273 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/adcb1b2dbf34483c8066221722417f4a 2023-04-13 20:54:11,275 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2bb9566dc8ee4fa28aeb3fe77f3c1a92 2023-04-13 20:54:11,277 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/0dfb46ad61d74ddc85b628f3031847b9 2023-04-13 20:54:11,279 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/fe327acc7c054510b3d4e0731ff3b91d 2023-04-13 20:54:11,282 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8038f91c8b9644aab396138e704ca45c 2023-04-13 20:54:11,284 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/870358ea07274f96b4170114b51b4f53 2023-04-13 20:54:11,286 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e247f26c06ea41d597f96d3e8f731314 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/e247f26c06ea41d597f96d3e8f731314 2023-04-13 20:54:11,287 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/30713c0f043b406eb928703737509c69 2023-04-13 20:54:11,289 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/1254d27e8a534c01a27b347b78b69285 2023-04-13 20:54:11,291 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/2371b5668e9c4ba7a400d5620b3b60d0 2023-04-13 20:54:11,326 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/e9a5792d2ade4751836e7e4f8c81aa4c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8e876720cc924977add2ed6a72e45dab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0be99b581d1041fc9e6608726d35b54b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5f9b40c67a8d4e2880e108fb9ec30533, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5a17b8a8b9454ed2a82295c0d2aad108, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/2b4139ea619e44938f8cdd2f91508392, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f0b2505680434e4aa23ddd50cb4581ec, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4f290a8255064f0f9eeb767ce95677c2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb] to archive 2023-04-13 20:54:11,327 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:11,330 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cdba5971b81641f09dbb1d3fe855c48e 2023-04-13 20:54:11,332 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f47be67d373d49daaedcad81e51fbef6 2023-04-13 20:54:11,333 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/e9a5792d2ade4751836e7e4f8c81aa4c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/e9a5792d2ade4751836e7e4f8c81aa4c 2023-04-13 20:54:11,335 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/59a3d15eedbf4178bbd9edb5aa6c4a83 2023-04-13 20:54:11,336 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6144193ddd754afdbcac030f024cd58f 2023-04-13 20:54:11,338 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8e876720cc924977add2ed6a72e45dab to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8e876720cc924977add2ed6a72e45dab 2023-04-13 20:54:11,339 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/abd00b0889b84faaa3754dec8e0480c9 2023-04-13 20:54:11,341 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/969862800ab1489393bbf79248452c4a 2023-04-13 20:54:11,342 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/14ec556c6b7e4bb6b2841ae257f6091c 2023-04-13 20:54:11,343 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0be99b581d1041fc9e6608726d35b54b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0be99b581d1041fc9e6608726d35b54b 2023-04-13 20:54:11,344 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/fc64db17f27846a3a021b65b425d3f51 2023-04-13 20:54:11,346 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/41a2e4d476454df7bea50b61e20477c1 2023-04-13 20:54:11,347 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5f9b40c67a8d4e2880e108fb9ec30533 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5f9b40c67a8d4e2880e108fb9ec30533 2023-04-13 20:54:11,349 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e4cdf1908ac43aabeef73827680018e 2023-04-13 20:54:11,350 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/eebb972a76d24868a6143e0ac873e118 2023-04-13 20:54:11,351 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5a17b8a8b9454ed2a82295c0d2aad108 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5a17b8a8b9454ed2a82295c0d2aad108 2023-04-13 20:54:11,353 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d4b8bf18f2624f6585ee3d5458c33c90 2023-04-13 20:54:11,354 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/c21868e8539a4d56b14dacdcdd6ba597 2023-04-13 20:54:11,355 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b4eb3af7db0d4a42a62daf72041e2a4f 2023-04-13 20:54:11,356 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/2b4139ea619e44938f8cdd2f91508392 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/2b4139ea619e44938f8cdd2f91508392 2023-04-13 20:54:11,357 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4a1880fb0f714b28913f149b5851886e 2023-04-13 20:54:11,359 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/d124c6ec7cc645029d028151f67d5280 2023-04-13 20:54:11,360 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/6ce2bd20996a4ffca04e295281cf1e85 2023-04-13 20:54:11,361 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/a4780ac045c641af81cf75ac4efae411 2023-04-13 20:54:11,362 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f0b2505680434e4aa23ddd50cb4581ec to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/f0b2505680434e4aa23ddd50cb4581ec 2023-04-13 20:54:11,363 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/9e2b211bb78c45eaa135b56414145a63 2023-04-13 20:54:11,365 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/8bb3da5c4e694039865794d9f0e9f148 2023-04-13 20:54:11,366 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/048816b4c69e4944989b5359526c9239 2023-04-13 20:54:11,367 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3cbfdc8c958c4f7f94dc739533a49ef0 2023-04-13 20:54:11,368 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/b149eb8f95cc45f5aa7b413d9dca479d 2023-04-13 20:54:11,369 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cc48b491bef643cdabd37e339bd8119c 2023-04-13 20:54:11,370 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/df51591efeb24ca18a0354f0ff14b2bc 2023-04-13 20:54:11,371 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/0e015a502d2d4c27b1fcfceb13ec1ecb 2023-04-13 20:54:11,373 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4f290a8255064f0f9eeb767ce95677c2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4f290a8255064f0f9eeb767ce95677c2 2023-04-13 20:54:11,374 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/4e20d276f9d447569e9230cb10b04c1e 2023-04-13 20:54:11,377 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/cb32c6dddb56469b92e6615979eb8011 2023-04-13 20:54:11,378 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/7c990624829f4f6d8a5d84d8ddb8aa95 2023-04-13 20:54:11,380 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/791926c773574742b87890c0363dd7fb 2023-04-13 20:54:11,385 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e95f458bad524ddb83a9f7b3f830da66, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/5b80d2f09f4c409593a55c2c5a52dae6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7ade46047be84656a6658d488dc2dee6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/039b50fe2d12463e9e2f1ea9d372122d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/31437c8bd27e4a929a6a5c6f7a674093, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e33b6c87d2744eb49fc48c72ae9ec7da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3f3d337e902f4fb3bf50c196586acf5c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3b86d80c66634bce9ea518ac638f328a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c] to archive 2023-04-13 20:54:11,386 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:11,388 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/a21d5e10fb044536b389c1ba26daa828 2023-04-13 20:54:11,390 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/75c3022945be471697fef4c24035d6e4 2023-04-13 20:54:11,391 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e95f458bad524ddb83a9f7b3f830da66 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e95f458bad524ddb83a9f7b3f830da66 2023-04-13 20:54:11,392 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/def0aea8f60d4b319daa37701c1ab872 2023-04-13 20:54:11,393 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8a0ad4272547e4895e3c702d80fb0e 2023-04-13 20:54:11,394 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/5b80d2f09f4c409593a55c2c5a52dae6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/5b80d2f09f4c409593a55c2c5a52dae6 2023-04-13 20:54:11,396 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8d4a3a08bac441b7acf3465d0bd755e9 2023-04-13 20:54:11,397 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/93cc6a7330a943219576a298ffc50f99 2023-04-13 20:54:11,398 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8f07cadf0678450f97966d2ad772f34e 2023-04-13 20:54:11,399 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7ade46047be84656a6658d488dc2dee6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7ade46047be84656a6658d488dc2dee6 2023-04-13 20:54:11,400 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/8965b5a537df4fafb6f36e71eb10db46 2023-04-13 20:54:11,401 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/43392edb255847a4ae75ae53671a9bba 2023-04-13 20:54:11,402 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/039b50fe2d12463e9e2f1ea9d372122d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/039b50fe2d12463e9e2f1ea9d372122d 2023-04-13 20:54:11,403 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f018d7dd850940e593d4c048144ce90d 2023-04-13 20:54:11,404 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/bdf2353039684be595121fa94d948bcc 2023-04-13 20:54:11,405 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/31437c8bd27e4a929a6a5c6f7a674093 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/31437c8bd27e4a929a6a5c6f7a674093 2023-04-13 20:54:11,406 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/7de3e4ed0d3845bd80b23e26ad2e6b66 2023-04-13 20:54:11,407 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/76180ae7a37d417da9c9233f5ad4de71 2023-04-13 20:54:11,408 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/951ac88e500b40e08e7678224a870bab 2023-04-13 20:54:11,410 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e33b6c87d2744eb49fc48c72ae9ec7da to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/e33b6c87d2744eb49fc48c72ae9ec7da 2023-04-13 20:54:11,411 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/639e5d91b1594761b79141700939255c 2023-04-13 20:54:11,412 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/d7759296146d4979bc14838545d47192 2023-04-13 20:54:11,413 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/9c8e08f7b1a8416bae6ed271173377b9 2023-04-13 20:54:11,414 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/dc2b3d893fc3488f9425ad70d5145aac 2023-04-13 20:54:11,415 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3f3d337e902f4fb3bf50c196586acf5c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3f3d337e902f4fb3bf50c196586acf5c 2023-04-13 20:54:11,416 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/345617486f7b4fd39aed5cc97b3ad118 2023-04-13 20:54:11,417 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/03e29cda01cb46119c59def3c4d899cc 2023-04-13 20:54:11,418 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/320a9e08fb564a6fb0ef3550d497571a 2023-04-13 20:54:11,419 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/23cab79b653a4899bcfc448aa0694c38 2023-04-13 20:54:11,420 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/39c9b11d840d42abbc2357721e22d75b 2023-04-13 20:54:11,421 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/f275c20ff65a4ff28f7ce9d9faf13410 2023-04-13 20:54:11,422 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6f1c9e6db12a41d597404557a0f5ca57 2023-04-13 20:54:11,423 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/c8a14579c96b45f2be0cc1630eeb0a92 2023-04-13 20:54:11,425 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3b86d80c66634bce9ea518ac638f328a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/3b86d80c66634bce9ea518ac638f328a 2023-04-13 20:54:11,426 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/15c1796ad56f4350869bc299ee693476 2023-04-13 20:54:11,427 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/1abd6e1bb9f34235b6c871ee08d576fe 2023-04-13 20:54:11,428 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/300216b435694d75a72df3228fd24f1f 2023-04-13 20:54:11,429 DEBUG [StoreCloser-TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/59013771fdef414a9794bcd9450c052c 2023-04-13 20:54:11,435 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/recovered.edits/620.seqid, newMaxSeqId=620, maxSeqId=1 2023-04-13 20:54:11,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb. 2023-04-13 20:54:11,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cdf6485ca98235662ec10d4d33684acb: 2023-04-13 20:54:11,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:11,439 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cdf6485ca98235662ec10d4d33684acb, regionState=CLOSED 2023-04-13 20:54:11,439 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419251439"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419251439"}]},"ts":"1681419251439"} 2023-04-13 20:54:11,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-13 20:54:11,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure cdf6485ca98235662ec10d4d33684acb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 497 msec 2023-04-13 20:54:11,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-13 20:54:11,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=cdf6485ca98235662ec10d4d33684acb, UNASSIGN in 501 msec 2023-04-13 20:54:11,445 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419251445"}]},"ts":"1681419251445"} 2023-04-13 20:54:11,446 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:54:11,448 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:54:11,450 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 519 msec 2023-04-13 20:54:11,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-13 20:54:11,541 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-13 20:54:11,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:54:11,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,553 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,555 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 20:54:11,559 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:11,562 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/recovered.edits] 2023-04-13 20:54:11,565 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8553bc46008b4cf0bc27567c0398618a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/8553bc46008b4cf0bc27567c0398618a 2023-04-13 20:54:11,567 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a6fb981cc87a4fd38b791a40033e34f9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/a6fb981cc87a4fd38b791a40033e34f9 2023-04-13 20:54:11,568 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f7f85f668efd423d86d064cd1579bb07 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/A/f7f85f668efd423d86d064cd1579bb07 2023-04-13 20:54:11,571 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3c7f1a6477774722945bd497daa9c017 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/3c7f1a6477774722945bd497daa9c017 2023-04-13 20:54:11,572 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5615a652f5b147babc16d1ed9cc72c04 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/5615a652f5b147babc16d1ed9cc72c04 2023-04-13 20:54:11,573 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/ccd39f9a00d946c2ab5354f8125f96dd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/B/ccd39f9a00d946c2ab5354f8125f96dd 2023-04-13 20:54:11,575 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/40e3a67d39904bdcb15ea2df436ae8a8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/40e3a67d39904bdcb15ea2df436ae8a8 2023-04-13 20:54:11,577 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6ec86191c77b4b299f512f365510b459 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/6ec86191c77b4b299f512f365510b459 2023-04-13 20:54:11,578 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/cd45548644894c20adcdee9294dd7f14 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/C/cd45548644894c20adcdee9294dd7f14 2023-04-13 20:54:11,582 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/recovered.edits/620.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb/recovered.edits/620.seqid 2023-04-13 20:54:11,583 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/cdf6485ca98235662ec10d4d33684acb 2023-04-13 20:54:11,583 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:54:11,585 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,605 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:54:11,652 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:54:11,654 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,654 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:54:11,654 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419251654"}]},"ts":"9223372036854775807"} 2023-04-13 20:54:11,656 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:54:11,656 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cdf6485ca98235662ec10d4d33684acb, NAME => 'TestAcidGuarantees,,1681419220700.cdf6485ca98235662ec10d4d33684acb.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:54:11,656 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:54:11,656 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419251656"}]},"ts":"9223372036854775807"} 2023-04-13 20:54:11,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 20:54:11,658 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:54:11,660 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 114 msec 2023-04-13 20:54:11,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-13 20:54:11,858 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-13 20:54:11,876 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: (jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1187394675_15 at /127.0.0.1:59756 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2c6dc074-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:36431-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2c6dc074-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2c6dc074-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2c6dc074-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=548 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=383 (was 323) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=846 (was 1497) 2023-04-13 20:54:11,888 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=548, MaxFileDescriptor=60000, SystemLoadAverage=383, ProcessCount=170, AvailableMemoryMB=845 2023-04-13 20:54:11,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:54:11,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:54:11,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:11,893 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:54:11,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-13 20:54:11,894 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:54:11,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:11,897 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:11,898 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb empty. 2023-04-13 20:54:11,898 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:11,898 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:54:11,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:12,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:12,315 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:54:12,316 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 12f4a42416bb74102917a94867a06dfb, NAME => 'TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:54:12,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 12f4a42416bb74102917a94867a06dfb, disabling compactions & flushes 2023-04-13 20:54:12,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. after waiting 0 ms 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:12,723 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:12,723 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:12,726 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:54:12,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419252727"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419252727"}]},"ts":"1681419252727"} 2023-04-13 20:54:12,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:54:12,729 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:54:12,729 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419252729"}]},"ts":"1681419252729"} 2023-04-13 20:54:12,730 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:54:12,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, ASSIGN}] 2023-04-13 20:54:12,735 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, ASSIGN 2023-04-13 20:54:12,735 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:54:12,886 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:12,887 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419252886"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419252886"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419252886"}]},"ts":"1681419252886"} 2023-04-13 20:54:12,888 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:12,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:13,044 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 12f4a42416bb74102917a94867a06dfb, NAME => 'TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:54:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:54:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,044 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,045 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,047 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A 2023-04-13 20:54:13,047 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A 2023-04-13 20:54:13,047 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:13,047 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName A 2023-04-13 20:54:13,048 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:13,048 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,049 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B 2023-04-13 20:54:13,049 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B 2023-04-13 20:54:13,049 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:13,049 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName B 2023-04-13 20:54:13,050 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:13,050 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,051 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C 2023-04-13 20:54:13,051 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C 2023-04-13 20:54:13,051 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:13,051 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName C 2023-04-13 20:54:13,052 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:13,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:13,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,053 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:54:13,057 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:13,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:54:13,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 12f4a42416bb74102917a94867a06dfb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74023641, jitterRate=0.10303820669651031}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:54:13,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:13,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., pid=18, masterSystemTime=1681419253040 2023-04-13 20:54:13,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:13,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:13,062 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:13,062 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419253062"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419253062"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419253062"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419253062"}]},"ts":"1681419253062"} 2023-04-13 20:54:13,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-13 20:54:13,065 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 175 msec 2023-04-13 20:54:13,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-13 20:54:13,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, ASSIGN in 332 msec 2023-04-13 20:54:13,067 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:54:13,067 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419253067"}]},"ts":"1681419253067"} 2023-04-13 20:54:13,068 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:54:13,070 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:54:13,072 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-04-13 20:54:13,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-13 20:54:13,999 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-13 20:54:14,000 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0186ae08 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,006 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@324dc814, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,008 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:14,012 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47146, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:14,013 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:54:14,015 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35718, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:54:14,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:54:14,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:54:14,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:14,441 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:54:14,442 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:54:14,442 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:54:14,446 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 20:54:14,458 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, REOPEN/MOVE}] 2023-04-13 20:54:14,459 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, REOPEN/MOVE 2023-04-13 20:54:14,460 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:14,460 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419254460"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419254460"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419254460"}]},"ts":"1681419254460"} 2023-04-13 20:54:14,461 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:14,613 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 12f4a42416bb74102917a94867a06dfb, disabling compactions & flushes 2023-04-13 20:54:14,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. after waiting 0 ms 2023-04-13 20:54:14,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 20:54:14,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,619 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:14,619 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 12f4a42416bb74102917a94867a06dfb to self. 2023-04-13 20:54:14,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=CLOSED 2023-04-13 20:54:14,621 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419254621"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419254621"}]},"ts":"1681419254621"} 2023-04-13 20:54:14,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-13 20:54:14,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 161 msec 2023-04-13 20:54:14,625 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=true 2023-04-13 20:54:14,775 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:14,775 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419254775"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419254775"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419254775"}]},"ts":"1681419254775"} 2023-04-13 20:54:14,777 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:14,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 12f4a42416bb74102917a94867a06dfb, NAME => 'TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:54:14,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:54:14,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,936 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,937 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A 2023-04-13 20:54:14,938 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A 2023-04-13 20:54:14,938 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:14,940 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName A 2023-04-13 20:54:14,942 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:14,943 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,944 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B 2023-04-13 20:54:14,944 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B 2023-04-13 20:54:14,944 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:14,944 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName B 2023-04-13 20:54:14,945 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:14,945 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,946 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C 2023-04-13 20:54:14,946 DEBUG [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C 2023-04-13 20:54:14,946 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:14,946 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 12f4a42416bb74102917a94867a06dfb columnFamilyName C 2023-04-13 20:54:14,947 INFO [StoreOpener-12f4a42416bb74102917a94867a06dfb-1] regionserver.HStore(310): Store=12f4a42416bb74102917a94867a06dfb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:14,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:54:14,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:14,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 12f4a42416bb74102917a94867a06dfb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60691207, jitterRate=-0.09563054144382477}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:54:14,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:14,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., pid=23, masterSystemTime=1681419254929 2023-04-13 20:54:14,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:14,955 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:14,955 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419254955"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419254955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419254955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419254955"}]},"ts":"1681419254955"} 2023-04-13 20:54:14,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-13 20:54:14,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 179 msec 2023-04-13 20:54:14,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-13 20:54:14,960 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, REOPEN/MOVE in 500 msec 2023-04-13 20:54:14,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-13 20:54:14,962 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 516 msec 2023-04-13 20:54:14,964 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 938 msec 2023-04-13 20:54:14,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-13 20:54:14,973 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x33a47c05 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,978 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7bbdba17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,978 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x21265fcb to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,982 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24bd3872, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,982 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x69f15cb4 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,985 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7aba8e46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,986 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0feb770d to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,992 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@210d1008, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,993 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x54451f11 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:14,997 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c2598b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:14,997 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x508246cc to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:15,002 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@421e5356, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:15,002 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x08795451 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:15,006 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a0fd7b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:15,006 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x58e27a32 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:15,010 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a7b66a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:15,010 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x092bed09 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:15,014 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42d325f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:15,019 DEBUG [hconnection-0xb5ab66b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,019 DEBUG [hconnection-0x752bab2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,020 DEBUG [hconnection-0x3c95254e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,020 DEBUG [hconnection-0x51c23be0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,021 DEBUG [hconnection-0x1afc6558-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,021 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,022 DEBUG [hconnection-0x50dbd12f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,022 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,025 DEBUG [hconnection-0x5b0b6bef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,028 DEBUG [hconnection-0x3cd421db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,028 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,028 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47170, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,028 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47186, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,028 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,031 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,031 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47224, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,028 DEBUG [hconnection-0x35dfa87a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:15,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,041 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:15,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:15,059 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:54:15,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:15,059 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:15,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:15,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:15,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:15,061 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:15,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:15,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:15,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:15,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:15,069 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,069 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:15,069 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:15,069 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:15,069 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:15,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:15,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:15,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b2d9dd6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:15,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:15,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,077 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:15,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:15,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:15,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,080 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:15,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:15,083 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:15,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:15,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:15,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:15,083 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:15,085 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,085 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,085 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:15,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:15,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:15,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,093 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:15,093 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:15,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419315090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419315091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419315093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419315099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,104 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413736ffd3fef8d4cef9147e4de6a7b50da to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413736ffd3fef8d4cef9147e4de6a7b50da 2023-04-13 20:54:15,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419315099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,137 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/d6473826a1e244cf8593d164ea3f8a49 2023-04-13 20:54:15,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/1e10ba8a585941b9a5d5409101c239b6 2023-04-13 20:54:15,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/af6ca7b777044efe939adc958edb595b 2023-04-13 20:54:15,193 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:15,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:15,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/d6473826a1e244cf8593d164ea3f8a49 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49 2023-04-13 20:54:15,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49, entries=200, sequenceid=17, filesize=31.9 K 2023-04-13 20:54:15,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/1e10ba8a585941b9a5d5409101c239b6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6 2023-04-13 20:54:15,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:15,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:15,205 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:15,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:15,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:15,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 20:54:15,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/af6ca7b777044efe939adc958edb595b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b 2023-04-13 20:54:15,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419315201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:15,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419315202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:15,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:15,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419315203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419315208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:15,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:15,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:15,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:15,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b, entries=150, sequenceid=17, filesize=11.6 K 2023-04-13 20:54:15,217 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 12f4a42416bb74102917a94867a06dfb in 158ms, sequenceid=17, compaction requested=false 2023-04-13 20:54:15,217 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:15,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419315203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,220 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:15,220 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:15,220 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,220 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:15,220 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:54:15,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:15,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:15,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,221 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:15,222 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,279 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c26794c9d4ab4a52b567fc86bcfd02c9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c26794c9d4ab4a52b567fc86bcfd02c9 2023-04-13 20:54:15,304 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/9fc16ef337124c00ad19f89d027ddcd9 2023-04-13 20:54:15,321 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/b86efdc127e54778b02ddebea854a149 2023-04-13 20:54:15,349 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/83f8f6a623fb4c989d55ce11d9973dad 2023-04-13 20:54:15,358 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/9fc16ef337124c00ad19f89d027ddcd9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9 2023-04-13 20:54:15,364 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9, entries=150, sequenceid=36, filesize=25.1 K 2023-04-13 20:54:15,365 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/b86efdc127e54778b02ddebea854a149 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149 2023-04-13 20:54:15,374 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 20:54:15,377 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/83f8f6a623fb4c989d55ce11d9973dad as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad 2023-04-13 20:54:15,385 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad, entries=150, sequenceid=36, filesize=11.6 K 2023-04-13 20:54:15,386 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 12f4a42416bb74102917a94867a06dfb in 166ms, sequenceid=36, compaction requested=false 2023-04-13 20:54:15,386 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:15,386 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,386 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:15,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:15,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:15,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f26d27[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:15,400 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:15,400 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:15,402 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,402 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:15,403 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:15,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:15,407 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:15,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:15,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:15,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:15,407 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:15,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:15,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:15,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,418 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:15,418 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:15,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:15,468 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:15,469 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:15,469 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:15,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,470 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:15,470 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:15,495 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041371a3ce88feec438095b789d58739b6ba to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041371a3ce88feec438095b789d58739b6ba 2023-04-13 20:54:15,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419315500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,518 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,518 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:15,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419315501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419315502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:15,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419315512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419315514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:15,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:15,540 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:15,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:15,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:15,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:15,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:15,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:15,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:15,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:15,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:15,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:15,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:15,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:15,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,555 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:15,555 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:15,555 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:15,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:15,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:15,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:15,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb95951[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:15,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:15,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:15,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:15,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:15,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:15,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:15,572 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:15,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:15,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:15,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:15,572 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:15,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,581 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,581 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,581 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,581 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:15,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:15,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:15,582 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:15,582 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:15,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419315619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419315619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419315622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419315622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,623 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419315623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,698 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:15,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:15,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:15,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:15,706 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:15,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:15,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:15,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:15,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:15,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:15,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:15,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:15,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:15,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:15,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eee5273[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:15,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:15,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,722 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:15,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:15,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:15,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:15,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:15,729 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:15,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:15,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:15,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:15,729 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:15,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,735 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:15,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:15,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:15,738 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,738 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,739 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:15,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419315822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419315822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419315825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419315825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:15,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419315826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,839 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:15,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:15,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:15,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:15,848 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:15,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:15,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:15,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:15,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:15,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:15,861 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,861 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:15,861 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:15,861 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:15,861 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:15,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@693ab355[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:15,866 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:15,866 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,867 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:15,867 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:15,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:15,869 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:15,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:15,873 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:15,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:15,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:15,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:15,873 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:15,874 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,874 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,874 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:15,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:15,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,876 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:15,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:15,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:15,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:15,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:15,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:15,881 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:15,881 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:15,884 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:15,884 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:15,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:15,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:15,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:15,913 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/0edde9b536f1493a9a33da8716ed05f4 2023-04-13 20:54:15,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/17a342b9e41b4d91829364fa5cff299e 2023-04-13 20:54:15,981 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:15,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:15,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:15,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/42c6bab0f6e945c88678af0f9c8e42a3 2023-04-13 20:54:16,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/0edde9b536f1493a9a33da8716ed05f4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4 2023-04-13 20:54:16,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,008 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4, entries=450, sequenceid=48, filesize=65.8 K 2023-04-13 20:54:16,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/17a342b9e41b4d91829364fa5cff299e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e 2023-04-13 20:54:16,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:16,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:16,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:16,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ff77000[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:16,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e, entries=150, sequenceid=48, filesize=11.6 K 2023-04-13 20:54:16,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:16,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/42c6bab0f6e945c88678af0f9c8e42a3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3 2023-04-13 20:54:16,020 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:16,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:16,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:16,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3, entries=150, sequenceid=48, filesize=11.6 K 2023-04-13 20:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:16,025 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 12f4a42416bb74102917a94867a06dfb in 558ms, sequenceid=48, compaction requested=true 2023-04-13 20:54:16,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:16,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:16,027 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:16,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:16,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:54:16,027 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,028 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 125826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,028 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:16,028 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:16,029 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:16,029 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,029 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,029 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=34.8 K 2023-04-13 20:54:16,029 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=122.9 K 2023-04-13 20:54:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,030 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d6473826a1e244cf8593d164ea3f8a49, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681419255047 2023-04-13 20:54:16,030 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1e10ba8a585941b9a5d5409101c239b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681419255047 2023-04-13 20:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:16,030 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:16,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:16,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:16,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:16,030 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:16,032 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9fc16ef337124c00ad19f89d027ddcd9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681419255077 2023-04-13 20:54:16,032 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b86efdc127e54778b02ddebea854a149, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681419255077 2023-04-13 20:54:16,032 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0edde9b536f1493a9a33da8716ed05f4, keycount=450, bloomtype=ROW, size=65.8 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:16,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,034 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 17a342b9e41b4d91829364fa5cff299e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:16,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,040 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:16,040 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,066 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:16,070 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#132 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:16,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419255491, max timestamp=1681419256124Num uniques 859; 2023-04-13 20:54:16,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:16,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419255491, max timestamp=1681419256126Num uniques 859; 2023-04-13 20:54:16,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:16,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419255491, max timestamp=1681419256126Num uniques 859; 2023-04-13 20:54:16,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:16,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:16,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:16,136 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:16,141 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:16,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:16,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:16,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,147 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419316153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419316153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419316154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419316159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419316160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,165 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,165 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:16,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:16,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4972d57e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:16,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:16,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:16,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:16,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:16,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:16,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:16,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:16,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:16,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:16,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:16,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:16,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,188 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:16,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,188 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:16,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:16,192 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c9e471bb762244778cb3fb8b86dec3ef to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c9e471bb762244778cb3fb8b86dec3ef 2023-04-13 20:54:16,213 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/d667c7cbbccb4e6ea6594f8ce1d21d09 2023-04-13 20:54:16,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419316261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419316261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419316263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419316267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419316267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,289 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:16,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:16,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:16,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:16,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:16,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45037be9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:16,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:16,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:16,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:16,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:16,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:16,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:16,311 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:16,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:16,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:16,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:16,311 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:16,312 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,312 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,312 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:16,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:16,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,319 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:16,319 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:16,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:16,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,419 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:16,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:16,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:16,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,423 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,425 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:16,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791fd807[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:16,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:16,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,434 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:16,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:16,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:16,436 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:16,445 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:16,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:16,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:16,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:16,446 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:16,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,451 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,454 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,460 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:16,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:16,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,461 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:16,461 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:16,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419316464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419316465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419316467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419316470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419316470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,491 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/c606210c3163433fbf98eb45bd08b7a3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/c606210c3163433fbf98eb45bd08b7a3 2023-04-13 20:54:16,499 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into c606210c3163433fbf98eb45bd08b7a3(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:16,499 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:16,500 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419256027; duration=0sec 2023-04-13 20:54:16,500 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:16,500 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:16,501 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:16,501 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:16,501 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,501 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=34.8 K 2023-04-13 20:54:16,502 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting af6ca7b777044efe939adc958edb595b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1681419255047 2023-04-13 20:54:16,502 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 83f8f6a623fb4c989d55ce11d9973dad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1681419255077 2023-04-13 20:54:16,503 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 42c6bab0f6e945c88678af0f9c8e42a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:16,512 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#136 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:16,561 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:16,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:16,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,569 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,569 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,569 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:16,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:16,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:16,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:16,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7decd054[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:16,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:16,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,579 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:16,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:16,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:16,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:16,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:16,583 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:16,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:16,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:16,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:16,583 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:16,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:16,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:16,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:16,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:16,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:16,589 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:16,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,590 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:16,590 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:16,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:16,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:16,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:16,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/8c1036d64584489e89fea5f53ed914ff 2023-04-13 20:54:16,657 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/992c8d3abb134bec91bb0589585d1906 2023-04-13 20:54:16,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/d667c7cbbccb4e6ea6594f8ce1d21d09 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09 2023-04-13 20:54:16,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09, entries=150, sequenceid=75, filesize=25.1 K 2023-04-13 20:54:16,668 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/8c1036d64584489e89fea5f53ed914ff as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff 2023-04-13 20:54:16,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff, entries=150, sequenceid=75, filesize=11.6 K 2023-04-13 20:54:16,674 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/992c8d3abb134bec91bb0589585d1906 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906 2023-04-13 20:54:16,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906, entries=150, sequenceid=75, filesize=11.6 K 2023-04-13 20:54:16,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~185.72 KB/190176, currentSize=114.05 KB/116790 for 12f4a42416bb74102917a94867a06dfb in 544ms, sequenceid=75, compaction requested=false 2023-04-13 20:54:16,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:16,690 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:16,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:16,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:16,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:16,700 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:16,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:16,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:16,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:16,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:16,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:16,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:16,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:16,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:16,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:16,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:16,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:16,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:16,709 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:16,709 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:16,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:16,710 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:16,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:16,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:16,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419316780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419316781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419316782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419316784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419316784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419316885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419316885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419316886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419316889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:16,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419316889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:16,913 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/336caaf9e491429f9e1e15a775825b8d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/336caaf9e491429f9e1e15a775825b8d 2023-04-13 20:54:16,924 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into 336caaf9e491429f9e1e15a775825b8d(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:16,925 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:16,925 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419256027; duration=0sec 2023-04-13 20:54:16,925 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:16,926 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e 2023-04-13 20:54:16,932 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into 187c4f0f6e9f4b3b9bfd26dc1b875c2e(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:16,932 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:16,932 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=13, startTime=1681419256027; duration=0sec 2023-04-13 20:54:16,932 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:16,985 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:54:17,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419317090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419317090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419317091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419317091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419317091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,143 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041355fd9bef4bf04d9abbeb757f9b80bbd1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041355fd9bef4bf04d9abbeb757f9b80bbd1 2023-04-13 20:54:17,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419317394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419317398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419317398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419317399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419317400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,551 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=96, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/441183c1dab9417faee20903acd94ed7 2023-04-13 20:54:17,566 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/eda7af6a1dc441bb9cce17cb4e32ede5 2023-04-13 20:54:17,577 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/65159487ae5241798be2594e40bb52ac 2023-04-13 20:54:17,581 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/441183c1dab9417faee20903acd94ed7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7 2023-04-13 20:54:17,585 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7, entries=150, sequenceid=96, filesize=25.1 K 2023-04-13 20:54:17,586 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/eda7af6a1dc441bb9cce17cb4e32ede5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5 2023-04-13 20:54:17,591 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5, entries=150, sequenceid=96, filesize=11.6 K 2023-04-13 20:54:17,592 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/65159487ae5241798be2594e40bb52ac as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac 2023-04-13 20:54:17,596 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac, entries=150, sequenceid=96, filesize=11.6 K 2023-04-13 20:54:17,597 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 12f4a42416bb74102917a94867a06dfb in 888ms, sequenceid=96, compaction requested=true 2023-04-13 20:54:17,597 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:17,598 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:17,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:17,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:17,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:17,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:17,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,599 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:17,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:17,600 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,600 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:17,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:17,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:17,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226a33b2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:17,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:17,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,602 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:17,602 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:17,603 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:17,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:17,604 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:17,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:17,604 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:17,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:17,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:17,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:17,607 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:17,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:17,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:17,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:17,607 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:17,610 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,611 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:17,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:17,611 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,612 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:17,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:17,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:17,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:17,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:17,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:17,614 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:17,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:17,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:17,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:17,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:17,615 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:17,615 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:17,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:17,715 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:17,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:17,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:17,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:17,718 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:17,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:17,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:17,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:17,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:17,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:17,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:17,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:17,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:17,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:17,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:17,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:17,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:17,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:17,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:17,722 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:17,722 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:17,722 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:17,723 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:17,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:17,902 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:17,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419317920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681419257905, max timestamp=1681419257920Num uniques 859; 2023-04-13 20:54:17,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:54:17,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681419257905, max timestamp=1681419257920Num uniques 859; 2023-04-13 20:54:17,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681419257905, max timestamp=1681419257920Num uniques 859; 2023-04-13 20:54:17,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:54:17,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:54:17,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419317938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419317939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419317940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:17,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:17,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419317945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419318024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419318046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419318047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419318048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419318051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,140 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f5457c8c554b470db6780c118b6b8817 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5457c8c554b470db6780c118b6b8817 2023-04-13 20:54:18,146 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/36b7f4153b524becb3969f27fbc046bd 2023-04-13 20:54:18,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419318229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419318249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419318251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419318251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419318254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419318533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419318552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419318553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419318555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,560 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/4c9bb471dabe44e887dcb3416ab52554 2023-04-13 20:54:18,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:18,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419318558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,972 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/a3690bf020d244a6881a430175820e21 2023-04-13 20:54:18,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/36b7f4153b524becb3969f27fbc046bd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd 2023-04-13 20:54:18,981 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd, entries=150, sequenceid=110, filesize=25.1 K 2023-04-13 20:54:18,983 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/4c9bb471dabe44e887dcb3416ab52554 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554 2023-04-13 20:54:18,987 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554, entries=150, sequenceid=110, filesize=11.6 K 2023-04-13 20:54:18,988 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/a3690bf020d244a6881a430175820e21 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21 2023-04-13 20:54:18,993 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21, entries=150, sequenceid=110, filesize=11.6 K 2023-04-13 20:54:18,994 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 12f4a42416bb74102917a94867a06dfb in 1272ms, sequenceid=110, compaction requested=true 2023-04-13 20:54:18,994 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:18,994 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:18,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,000 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68c825[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,005 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,008 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,008 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,016 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,017 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,017 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:19,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:19,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419319060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419319061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419319062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419319064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419319064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,117 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,121 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,123 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,123 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,124 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,124 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,125 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,125 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,126 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b30f84f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,129 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,129 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,129 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,131 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,134 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,134 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,138 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,144 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,144 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419319165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419319166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419319168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,244 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,247 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,254 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c5f90d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,258 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,258 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,258 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,258 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,261 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,261 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,264 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,268 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,268 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,368 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,371 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419319368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419319369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,374 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419319371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,374 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,374 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,374 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,375 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,376 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,376 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,376 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,376 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,378 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,378 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c1a104[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,379 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,379 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,382 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,385 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,386 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,388 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,389 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,395 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,396 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,463 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413a9ef413f662c4a968a93ab15d08f843e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a9ef413f662c4a968a93ab15d08f843e 2023-04-13 20:54:19,469 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/53213e4478f74be3969f1e92e11efb54 2023-04-13 20:54:19,496 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,499 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,500 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,500 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,501 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,503 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,503 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,503 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,503 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,504 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,504 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,505 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,505 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29d27ab9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,507 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,507 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,509 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,509 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,509 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,509 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,509 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,509 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,513 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,514 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,517 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,521 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,521 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,621 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,625 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,630 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,630 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@555c1d4c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,637 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,640 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,640 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,640 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,640 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,640 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,643 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,643 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,649 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,650 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,650 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,676 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419319674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419319675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419319675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,750 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,754 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,755 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,757 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,758 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,758 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,759 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,759 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1e9b7b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,763 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,767 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,767 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,767 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,767 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,767 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,769 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,774 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,775 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,875 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:19,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:19,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:19,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:19,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:19,878 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:19,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:19,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:19,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/17829bbf330d4d438682e5b599f557b6 2023-04-13 20:54:19,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:19,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:19,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:19,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:19,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:19,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:19,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:19,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:19,888 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:19,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449c0bd[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:19,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:19,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,893 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:19,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:19,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:19,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:19,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:19,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:19,898 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:19,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:19,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:19,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:19,898 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:19,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:19,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:19,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:19,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:19,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:19,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:19,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:19,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:19,905 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:19,906 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:19,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:19,906 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:19,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,006 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,009 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:20,014 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e74753f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,020 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,024 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,025 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,028 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,036 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,036 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419320068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:20,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419320070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,136 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,140 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,146 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:20,146 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf7b9c0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,152 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,160 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,160 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,169 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,169 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:20,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419320177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:20,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419320179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:20,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419320180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,269 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,273 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,277 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:20,277 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,278 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655c3e86[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,291 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,291 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/7da124cff724409183bc11aec38cfa02 2023-04-13 20:54:20,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,300 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/53213e4478f74be3969f1e92e11efb54 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54 2023-04-13 20:54:20,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,302 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,302 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54, entries=200, sequenceid=145, filesize=32.1 K 2023-04-13 20:54:20,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/17829bbf330d4d438682e5b599f557b6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6 2023-04-13 20:54:20,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6, entries=150, sequenceid=145, filesize=11.8 K 2023-04-13 20:54:20,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/7da124cff724409183bc11aec38cfa02 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02 2023-04-13 20:54:20,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02, entries=150, sequenceid=145, filesize=11.8 K 2023-04-13 20:54:20,316 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 12f4a42416bb74102917a94867a06dfb in 1273ms, sequenceid=145, compaction requested=true 2023-04-13 20:54:20,316 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:20,316 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:20,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:20,316 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:54:20,316 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:20,318 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135973 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:20,318 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:20,318 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59683 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:20,318 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,318 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/336caaf9e491429f9e1e15a775825b8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=132.8 K 2023-04-13 20:54:20,318 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:20,318 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,318 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/c606210c3163433fbf98eb45bd08b7a3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=58.3 K 2023-04-13 20:54:20,319 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 336caaf9e491429f9e1e15a775825b8d, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:20,319 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c606210c3163433fbf98eb45bd08b7a3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:20,319 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d667c7cbbccb4e6ea6594f8ce1d21d09, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681419255500 2023-04-13 20:54:20,319 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8c1036d64584489e89fea5f53ed914ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681419255500 2023-04-13 20:54:20,319 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 441183c1dab9417faee20903acd94ed7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681419256153 2023-04-13 20:54:20,320 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting eda7af6a1dc441bb9cce17cb4e32ede5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681419256153 2023-04-13 20:54:20,320 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 36b7f4153b524becb3969f27fbc046bd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681419256781 2023-04-13 20:54:20,321 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4c9bb471dabe44e887dcb3416ab52554, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681419256781 2023-04-13 20:54:20,321 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 53213e4478f74be3969f1e92e11efb54, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681419257936 2023-04-13 20:54:20,322 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 17829bbf330d4d438682e5b599f557b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681419257936 2023-04-13 20:54:20,333 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#148 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:20,336 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#147 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:20,402 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,407 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,414 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,414 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,415 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:20,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:20,444 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c7b2d4313e894968a9a9ce78ea6b028c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c7b2d4313e894968a9a9ce78ea6b028c 2023-04-13 20:54:20,458 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/b1a7b2063bfc426cb7abd98d6bb424ea 2023-04-13 20:54:20,478 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/96fc3365a55a402dba974bdb91f45fcb 2023-04-13 20:54:20,494 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/14d0751ae4534967a7633cf0357ce3e2 2023-04-13 20:54:20,499 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/b1a7b2063bfc426cb7abd98d6bb424ea as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea 2023-04-13 20:54:20,503 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea, entries=150, sequenceid=156, filesize=25.3 K 2023-04-13 20:54:20,505 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/96fc3365a55a402dba974bdb91f45fcb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb 2023-04-13 20:54:20,512 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb, entries=150, sequenceid=156, filesize=11.8 K 2023-04-13 20:54:20,513 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/14d0751ae4534967a7633cf0357ce3e2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2 2023-04-13 20:54:20,521 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2, entries=150, sequenceid=156, filesize=11.8 K 2023-04-13 20:54:20,522 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 12f4a42416bb74102917a94867a06dfb in 108ms, sequenceid=156, compaction requested=true 2023-04-13 20:54:20,522 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,522 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,524 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5a56b8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,543 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,544 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,547 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,550 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,551 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,551 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,651 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,672 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,677 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,686 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@321a03e1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,693 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,694 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,694 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,695 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,698 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,699 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,704 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,707 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,707 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,758 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/ed77dd54c39e42238f49b7ccab77e60b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ed77dd54c39e42238f49b7ccab77e60b 2023-04-13 20:54:20,764 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into ed77dd54c39e42238f49b7ccab77e60b(size=12.0 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:20,764 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,764 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=11, startTime=1681419260316; duration=0sec 2023-04-13 20:54:20,764 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:20,764 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:54:20,766 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71719 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:54:20,766 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:20,767 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,767 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=70.0 K 2023-04-13 20:54:20,767 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 187c4f0f6e9f4b3b9bfd26dc1b875c2e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1681419255426 2023-04-13 20:54:20,768 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 992c8d3abb134bec91bb0589585d1906, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1681419255500 2023-04-13 20:54:20,768 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 65159487ae5241798be2594e40bb52ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1681419256153 2023-04-13 20:54:20,769 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a3690bf020d244a6881a430175820e21, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1681419256781 2023-04-13 20:54:20,769 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7da124cff724409183bc11aec38cfa02, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681419257936 2023-04-13 20:54:20,770 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 14d0751ae4534967a7633cf0357ce3e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681419259059 2023-04-13 20:54:20,781 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#152 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:20,795 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/a3b02e347c2248108839dbf48699afbe as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3b02e347c2248108839dbf48699afbe 2023-04-13 20:54:20,801 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into a3b02e347c2248108839dbf48699afbe(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:20,801 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,801 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=10, startTime=1681419260316; duration=0sec 2023-04-13 20:54:20,801 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:20,808 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,813 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,813 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,813 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,813 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,816 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,818 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,818 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,819 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,819 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,821 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49595db[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,827 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,830 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,830 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,841 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,842 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:20,942 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:20,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:20,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:20,946 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:20,947 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,947 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:20,947 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:20,947 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,947 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,948 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:20,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:20,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:20,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:20,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:20,950 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:20,951 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,951 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:20,951 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:20,951 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:20,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:20,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:20,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:20,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b933c1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:20,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,957 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:20,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:20,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:20,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:20,962 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:20,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:20,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:20,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:20,962 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:20,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:20,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,964 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:20,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:20,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:20,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:20,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:20,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:20,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:20,970 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:20,971 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,071 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,080 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dab5af8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,090 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,097 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,097 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,100 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,101 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,104 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,104 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,176 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/70cc38487fcd42ccaeee1e8232aabd04 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/70cc38487fcd42ccaeee1e8232aabd04 2023-04-13 20:54:21,182 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into 70cc38487fcd42ccaeee1e8232aabd04(size=25.6 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:21,182 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:21,182 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=11, startTime=1681419260316; duration=0sec 2023-04-13 20:54:21,182 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:21,205 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,209 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:21,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,218 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fcc5ba7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,222 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,222 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,225 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:21,227 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:21,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:21,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,230 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,230 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,232 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,232 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,232 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,233 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,245 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,245 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,245 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,245 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419321261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419321262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419321266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,345 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,349 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@375f2ec8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,360 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,361 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,364 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,364 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,364 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,364 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,364 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419321367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419321368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,371 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,372 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419321370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,372 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,472 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,476 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,478 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,481 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:21,481 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,481 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,484 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,485 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e3124d9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,487 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,489 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,489 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,492 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,492 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,495 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,495 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,495 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:21,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,501 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,501 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419321571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419321571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419321573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,601 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,605 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,609 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,609 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,609 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:21,609 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,609 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,611 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,611 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@596add7d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,617 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,621 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,621 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,621 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,622 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,623 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,623 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,631 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,632 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,632 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:21,656 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413c2eac6531b514f11817542da9a57ac4c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c2eac6531b514f11817542da9a57ac4c 2023-04-13 20:54:21,662 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/3f848fe1514b434ca15243373385576e 2023-04-13 20:54:21,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/2ca4ee9347c6439db2a32112a422a2ad 2023-04-13 20:54:21,733 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,736 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,737 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,738 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,738 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,738 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,739 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,744 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,744 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,744 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:21,744 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2406b494[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,755 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,756 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,756 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,760 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,762 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,762 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,863 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,867 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:21,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,870 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:21,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:21,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:21,871 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:21,871 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:21,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:21,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:21,872 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:21,873 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:21,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a61e7b2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:21,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:21,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,876 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:21,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:21,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,877 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419321874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419321875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,878 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:21,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419321879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:21,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:21,881 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:21,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:21,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:21,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:21,881 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:21,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:21,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:21,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:21,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:21,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,895 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:21,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:21,895 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:21,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:21,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:21,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:21,996 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:21,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:21,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:21,999 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:21,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:21,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:21,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,001 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,003 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:22,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,008 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0101cb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,019 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,023 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,027 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,027 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,028 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,028 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,036 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,036 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,088 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/ffd4b621af714e529f37065cb2739149 2023-04-13 20:54:22,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419322088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419322088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,091 DEBUG [Thread-919] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:22,093 DEBUG [Thread-921] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:22,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/3f848fe1514b434ca15243373385576e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e 2023-04-13 20:54:22,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e, entries=450, sequenceid=176, filesize=66.3 K 2023-04-13 20:54:22,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/2ca4ee9347c6439db2a32112a422a2ad as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad 2023-04-13 20:54:22,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad, entries=150, sequenceid=176, filesize=11.8 K 2023-04-13 20:54:22,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/ffd4b621af714e529f37065cb2739149 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149 2023-04-13 20:54:22,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149, entries=150, sequenceid=176, filesize=11.8 K 2023-04-13 20:54:22,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 12f4a42416bb74102917a94867a06dfb in 885ms, sequenceid=176, compaction requested=true 2023-04-13 20:54:22,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:22,112 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:22,112 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:22,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:22,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:22,113 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 119910 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:22,113 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:22,113 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:22,113 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,114 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:22,114 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/70cc38487fcd42ccaeee1e8232aabd04, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=117.1 K 2023-04-13 20:54:22,114 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,114 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ed77dd54c39e42238f49b7ccab77e60b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=35.5 K 2023-04-13 20:54:22,114 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 70cc38487fcd42ccaeee1e8232aabd04, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681419257936 2023-04-13 20:54:22,114 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ed77dd54c39e42238f49b7ccab77e60b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=145, earliestPutTs=1681419257936 2023-04-13 20:54:22,114 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b1a7b2063bfc426cb7abd98d6bb424ea, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681419259059 2023-04-13 20:54:22,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 96fc3365a55a402dba974bdb91f45fcb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681419259059 2023-04-13 20:54:22,115 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 3f848fe1514b434ca15243373385576e, keycount=450, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681419261216 2023-04-13 20:54:22,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2ca4ee9347c6439db2a32112a422a2ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681419261226 2023-04-13 20:54:22,124 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#157 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:22,129 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#156 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:22,136 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,144 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,144 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,144 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,144 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,164 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:22,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:22,165 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,176 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041317729bd7dbb44e7eaa061e0a0b374317 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041317729bd7dbb44e7eaa061e0a0b374317 2023-04-13 20:54:22,182 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=196, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/9abe58a978d74b0da42d4969ec974c97 2023-04-13 20:54:22,196 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/a38d32f999854e1c90d9e8ec993d25de 2023-04-13 20:54:22,206 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=196 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/1ded7a1a45f34d639e31aec26252d2d2 2023-04-13 20:54:22,210 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/9abe58a978d74b0da42d4969ec974c97 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97 2023-04-13 20:54:22,214 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97, entries=150, sequenceid=196, filesize=25.3 K 2023-04-13 20:54:22,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/a38d32f999854e1c90d9e8ec993d25de as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de 2023-04-13 20:54:22,218 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de, entries=150, sequenceid=196, filesize=11.8 K 2023-04-13 20:54:22,219 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/1ded7a1a45f34d639e31aec26252d2d2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2 2023-04-13 20:54:22,227 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2, entries=150, sequenceid=196, filesize=11.8 K 2023-04-13 20:54:22,228 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 12f4a42416bb74102917a94867a06dfb in 64ms, sequenceid=196, compaction requested=true 2023-04-13 20:54:22,228 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,228 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,232 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,232 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@120fa623[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,236 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,236 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,236 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,238 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,241 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,241 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,245 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,246 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,249 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,249 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,349 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,355 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,357 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,358 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,362 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,362 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731f7302[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,365 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,365 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,367 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,367 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,369 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,369 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,372 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,373 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,380 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,380 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,381 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:22,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:22,424 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:22,424 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:22,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:22,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:22,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419322453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419322454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,458 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413872e67f086064b9ab07f61ac6adcf948 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872e67f086064b9ab07f61ac6adcf948 2023-04-13 20:54:22,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419322455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,476 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/799bf261e8ce4f8a9a9bef600d4a17df 2023-04-13 20:54:22,481 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,485 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,485 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,485 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,485 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,487 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,487 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,489 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,489 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,492 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:22,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,502 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c4eedfc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,504 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,506 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,508 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,508 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,510 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,512 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,512 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,512 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,512 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,512 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,516 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,519 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,524 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,523 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,524 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,554 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/1464d01b07304f829133770470e464ed as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1464d01b07304f829133770470e464ed 2023-04-13 20:54:22,560 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into 1464d01b07304f829133770470e464ed(size=12.1 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:22,560 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,560 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419262112; duration=0sec 2023-04-13 20:54:22,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,560 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419322557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,561 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:22,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419322558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419322562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,564 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:22,564 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:22,564 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,564 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3b02e347c2248108839dbf48699afbe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=35.6 K 2023-04-13 20:54:22,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a3b02e347c2248108839dbf48699afbe, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1681419259059 2023-04-13 20:54:22,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ffd4b621af714e529f37065cb2739149, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681419261226 2023-04-13 20:54:22,565 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1ded7a1a45f34d639e31aec26252d2d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681419261259 2023-04-13 20:54:22,582 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#163 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:22,589 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/acb84ec90c3041bb982552b9a1a00513 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/acb84ec90c3041bb982552b9a1a00513 2023-04-13 20:54:22,595 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into acb84ec90c3041bb982552b9a1a00513(size=25.7 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:22,595 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,595 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419262112; duration=0sec 2023-04-13 20:54:22,595 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:22,609 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/600361f72c8e473888abae393209b760 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/600361f72c8e473888abae393209b760 2023-04-13 20:54:22,615 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into 600361f72c8e473888abae393209b760(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:22,615 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:22,615 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=13, startTime=1681419262112; duration=0sec 2023-04-13 20:54:22,615 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:22,624 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,628 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,628 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,632 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,632 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,633 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:22,633 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,635 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f865795[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,643 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,643 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,647 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,647 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,649 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:22,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,656 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,656 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,756 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,760 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,760 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,760 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,760 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,762 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,763 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:22,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419322762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419322762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419322766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3006ae0c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,768 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,768 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,769 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,770 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,772 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,772 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,773 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,775 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,775 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,777 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,778 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,781 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,781 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,881 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:22,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:22,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:22,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:22,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:22,884 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:22,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:22,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:22,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:22,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:22,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:22,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:22,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:22,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:22,888 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:22,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:22,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:22,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:22,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:22,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:22,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:22,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:22,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38895c6d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:22,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:22,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:22,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:22,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:22,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:22,899 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:22,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:22,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:22,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:22,899 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:22,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:22,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/6e89ea656c014e29b5394986aad86f1f 2023-04-13 20:54:22,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:22,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:22,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:22,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:22,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:22,910 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:22,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:22,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:22,910 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:22,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:22,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:22,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:23,010 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:23,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:23,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:23,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:23,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:23,014 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:23,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:23,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:23,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:23,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:23,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:23,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:23,019 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,019 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:23,019 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:23,019 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:23,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:23,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:23,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:23,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5343678d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:23,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,024 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:23,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:23,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:23,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:23,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:23,029 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:23,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:23,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:23,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:23,029 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:23,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:23,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:23,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,037 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:23,037 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:23,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:23,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419323068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419323068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419323068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,137 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:23,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:23,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:23,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:23,140 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:23,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:23,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:23,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:23,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:23,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:23,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:23,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:23,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb4a88f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:23,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:23,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:23,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:23,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,152 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:23,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:23,155 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:23,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:23,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:23,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:23,155 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:23,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:23,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,163 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:23,163 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:23,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,263 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:23,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:23,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:23,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:23,267 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:23,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:23,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:23,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:23,270 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,270 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:23,270 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:23,270 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:23,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:23,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:23,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b47039b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:23,274 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:23,274 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:23,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:23,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:23,280 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:23,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:23,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:23,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:23,280 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:23,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:23,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:23,287 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,287 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,287 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,287 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:23,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/d7d93b46bc0742658d09e179f468a2a8 2023-04-13 20:54:23,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/799bf261e8ce4f8a9a9bef600d4a17df as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df 2023-04-13 20:54:23,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df, entries=400, sequenceid=209, filesize=59.4 K 2023-04-13 20:54:23,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/6e89ea656c014e29b5394986aad86f1f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f 2023-04-13 20:54:23,360 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f, entries=150, sequenceid=209, filesize=11.8 K 2023-04-13 20:54:23,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/d7d93b46bc0742658d09e179f468a2a8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8 2023-04-13 20:54:23,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8, entries=150, sequenceid=209, filesize=11.8 K 2023-04-13 20:54:23,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 12f4a42416bb74102917a94867a06dfb in 947ms, sequenceid=209, compaction requested=true 2023-04-13 20:54:23,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:23,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:23,370 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:23,370 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:23,371 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:23,371 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:23,372 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113022 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:23,372 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:23,373 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,373 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/acb84ec90c3041bb982552b9a1a00513, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=110.4 K 2023-04-13 20:54:23,373 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36484 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:23,373 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:23,373 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,373 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1464d01b07304f829133770470e464ed, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=35.6 K 2023-04-13 20:54:23,373 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting acb84ec90c3041bb982552b9a1a00513, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681419261226 2023-04-13 20:54:23,374 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1464d01b07304f829133770470e464ed, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1681419261226 2023-04-13 20:54:23,374 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9abe58a978d74b0da42d4969ec974c97, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681419261259 2023-04-13 20:54:23,375 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a38d32f999854e1c90d9e8ec993d25de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681419261259 2023-04-13 20:54:23,375 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 799bf261e8ce4f8a9a9bef600d4a17df, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681419262394 2023-04-13 20:54:23,375 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6e89ea656c014e29b5394986aad86f1f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681419262394 2023-04-13 20:54:23,387 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:23,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:23,393 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#165 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:23,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:23,394 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:23,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:23,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:23,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:23,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:23,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:23,402 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#166 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:23,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:23,410 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:23,410 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,428 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413dee8c190d1324cd4a9f2b4e6c37bf7c1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413dee8c190d1324cd4a9f2b4e6c37bf7c1 2023-04-13 20:54:23,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:23,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:23,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419323579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419323580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419323580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419323683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419323683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47208 deadline: 1681419323683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,814 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/b18bac294abb4ea59568c9142d092755 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b18bac294abb4ea59568c9142d092755 2023-04-13 20:54:23,820 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into b18bac294abb4ea59568c9142d092755(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:23,820 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:23,820 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419263370; duration=0sec 2023-04-13 20:54:23,820 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:23,820 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 20:54:23,821 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:54:23,821 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:54:23,821 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. because compaction request was cancelled 2023-04-13 20:54:23,834 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/fbb9592466e04b259b0d4cbdd62e2106 2023-04-13 20:54:23,847 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/11564e03580c418197ef44bc79209e56 2023-04-13 20:54:23,866 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/5646ac41683647c28524abc5fe5bc34b 2023-04-13 20:54:23,871 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/fbb9592466e04b259b0d4cbdd62e2106 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106 2023-04-13 20:54:23,875 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106, entries=150, sequenceid=232, filesize=25.3 K 2023-04-13 20:54:23,876 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/11564e03580c418197ef44bc79209e56 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56 2023-04-13 20:54:23,884 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56, entries=150, sequenceid=232, filesize=11.8 K 2023-04-13 20:54:23,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/5646ac41683647c28524abc5fe5bc34b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b 2023-04-13 20:54:23,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419323885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419323885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,890 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b, entries=150, sequenceid=232, filesize=11.8 K 2023-04-13 20:54:23,891 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 12f4a42416bb74102917a94867a06dfb in 481ms, sequenceid=232, compaction requested=true 2023-04-13 20:54:23,891 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:23,891 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:23,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:23,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:23,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:23,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:23,898 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:23,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:23,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69df5eaa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:23,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:23,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:23,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:23,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:23,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:23,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:23,913 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:23,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:23,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:23,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:23,913 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:23,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:23,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,922 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:23,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:23,922 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:23,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:23,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:23,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:23,922 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:23,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:23,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:23,923 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:23,923 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:23,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:23,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419323965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,023 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,027 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,028 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,030 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21e83313[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,041 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,041 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,042 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,042 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,042 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,047 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,047 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419324068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,147 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,150 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,152 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,154 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,154 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,154 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,154 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a230771[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,164 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,164 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,180 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,180 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419324189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419324191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,230 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/da7bf5c8313042d592fbbf6216eeb814 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/da7bf5c8313042d592fbbf6216eeb814 2023-04-13 20:54:24,234 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into da7bf5c8313042d592fbbf6216eeb814(size=25.8 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:24,234 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:24,234 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419263370; duration=0sec 2023-04-13 20:54:24,234 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:24,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419324271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,280 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,283 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,287 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,287 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,287 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,287 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@395b9475[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,293 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,297 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,297 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,298 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,298 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,298 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,306 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,306 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,331 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041319f6c37a473d4b5889340a07e92f4415 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f6c37a473d4b5889340a07e92f4415 2023-04-13 20:54:24,406 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,410 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,413 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,415 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a62a7fa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,422 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,422 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,429 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,429 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,529 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,532 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,536 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,538 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af2690e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,546 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,546 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,553 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,554 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419324575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,654 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,657 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,659 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,661 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,661 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,661 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,661 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,662 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,662 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4868f7ae[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,671 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,671 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,673 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,673 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,673 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,674 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,678 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,678 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,678 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,678 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,678 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419324693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:24,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419324695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,739 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=247, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/11b0a94ce9c44457aabc8c8520abc778 2023-04-13 20:54:24,778 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,781 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,790 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,791 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,791 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,791 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,794 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2df52aea[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,799 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,801 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,802 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,806 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,810 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,810 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,910 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:24,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:24,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:24,914 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:24,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:24,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:24,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:24,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:24,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:24,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:24,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:24,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:24,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:24,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:24,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:24,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@121715d9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:24,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:24,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:24,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:24,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:24,928 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:24,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:24,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:24,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:24,929 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:24,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:24,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,931 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:24,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:24,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:24,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:24,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:24,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:24,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:24,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:24,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:24,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:24,935 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:24,935 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:24,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:24,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:24,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:24,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:25,035 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:25,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:25,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:25,047 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:25,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:25,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:25,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:25,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:25,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:25,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:25,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:25,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:25,054 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:25,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:25,055 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,055 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,055 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b74b48c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,059 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:25,059 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:25,059 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:25,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:25,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:25,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:25,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,060 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:25,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:25,063 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:25,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:25,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:25,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:25,063 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:25,069 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,069 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,069 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:25,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:25,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:25,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:25,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:25,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:25,079 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:25,079 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:25,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:25,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419325080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/376bde5929254e1cb1d0466b1c1ff0b3 2023-04-13 20:54:25,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/97fd8647ab444827a3dd778fb7f9ce5f 2023-04-13 20:54:25,179 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:25,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:25,185 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:25,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:25,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/11b0a94ce9c44457aabc8c8520abc778 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778 2023-04-13 20:54:25,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:25,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:25,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778, entries=200, sequenceid=247, filesize=32.1 K 2023-04-13 20:54:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:25,195 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,195 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:25,195 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:25,195 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:25,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/376bde5929254e1cb1d0466b1c1ff0b3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3 2023-04-13 20:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:25,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,203 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3, entries=150, sequenceid=247, filesize=11.8 K 2023-04-13 20:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442effcb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:25,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:25,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/97fd8647ab444827a3dd778fb7f9ce5f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f 2023-04-13 20:54:25,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f, entries=150, sequenceid=247, filesize=11.8 K 2023-04-13 20:54:25,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:25,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:25,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:25,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 12f4a42416bb74102917a94867a06dfb in 1313ms, sequenceid=247, compaction requested=true 2023-04-13 20:54:25,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:25,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:25,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:25,211 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:25,211 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:25,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:25,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:25,213 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:25,213 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85156 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:25,213 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:25,213 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:25,213 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,213 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,213 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b18bac294abb4ea59568c9142d092755, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=35.7 K 2023-04-13 20:54:25,213 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/da7bf5c8313042d592fbbf6216eeb814, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=83.2 K 2023-04-13 20:54:25,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:25,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:25,216 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:25,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:25,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:25,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:25,216 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:25,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:25,218 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting da7bf5c8313042d592fbbf6216eeb814, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681419262394 2023-04-13 20:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:25,218 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b18bac294abb4ea59568c9142d092755, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681419262394 2023-04-13 20:54:25,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:25,218 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:25,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,218 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fbb9592466e04b259b0d4cbdd62e2106, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681419262449 2023-04-13 20:54:25,219 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 11b0a94ce9c44457aabc8c8520abc778, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263573 2023-04-13 20:54:25,219 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 11564e03580c418197ef44bc79209e56, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681419262449 2023-04-13 20:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:25,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,220 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 376bde5929254e1cb1d0466b1c1ff0b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263575 2023-04-13 20:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:25,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,225 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:25,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:25,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:25,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:25,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,229 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:25,229 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:25,229 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:25,231 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#174 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:25,234 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#173 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:25,259 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/f27a33496dc843e48a6dab0f738f069f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/f27a33496dc843e48a6dab0f738f069f 2023-04-13 20:54:25,267 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into f27a33496dc843e48a6dab0f738f069f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:25,267 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:25,267 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419265211; duration=0sec 2023-04-13 20:54:25,268 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:25,268 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:25,271 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48554 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:25,271 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:25,271 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,271 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/600361f72c8e473888abae393209b760, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=47.4 K 2023-04-13 20:54:25,272 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 600361f72c8e473888abae393209b760, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=196, earliestPutTs=1681419261259 2023-04-13 20:54:25,272 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d7d93b46bc0742658d09e179f468a2a8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1681419262394 2023-04-13 20:54:25,273 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5646ac41683647c28524abc5fe5bc34b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1681419262449 2023-04-13 20:54:25,273 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 97fd8647ab444827a3dd778fb7f9ce5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263575 2023-04-13 20:54:25,286 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#175 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:25,324 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/5ea8f1f622b344518ddd41c35646acb0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/5ea8f1f622b344518ddd41c35646acb0 2023-04-13 20:54:25,326 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/cbaa9bf428c94f55b37a8ec4ede02adc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cbaa9bf428c94f55b37a8ec4ede02adc 2023-04-13 20:54:25,329 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:25,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:25,331 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into 5ea8f1f622b344518ddd41c35646acb0(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:25,331 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:25,331 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419265211; duration=0sec 2023-04-13 20:54:25,331 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:25,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:25,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:25,336 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:25,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:25,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:25,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,338 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:25,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:25,339 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,339 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:25,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:25,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:25,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:25,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:25,353 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:25,354 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:25,354 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:25,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:25,355 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:25,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:25,356 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:25,357 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into cbaa9bf428c94f55b37a8ec4ede02adc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:25,357 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:25,357 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=12, startTime=1681419265211; duration=0sec 2023-04-13 20:54:25,357 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:25,377 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b863d475ad9c49aa842dc546a62b18f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b863d475ad9c49aa842dc546a62b18f4 2023-04-13 20:54:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:25,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:25,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:25,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:25,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419325720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47162 deadline: 1681419325721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,784 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/7818d93af6504becb8441f539942b97c 2023-04-13 20:54:25,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:25,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419325827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:25,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:25,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419325827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419326029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47224 deadline: 1681419326030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419326087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47170 deadline: 1681419326102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,105 DEBUG [Thread-919] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8166 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:26,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47178 deadline: 1681419326126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,129 DEBUG [Thread-921] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:26,204 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/9a3cfb79db914c758c448019f51c4589 2023-04-13 20:54:26,216 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/0f9e638280024f1d90839dd0a78cc8c2 2023-04-13 20:54:26,220 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/7818d93af6504becb8441f539942b97c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c 2023-04-13 20:54:26,225 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c, entries=150, sequenceid=268, filesize=25.4 K 2023-04-13 20:54:26,225 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/9a3cfb79db914c758c448019f51c4589 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589 2023-04-13 20:54:26,236 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589, entries=150, sequenceid=268, filesize=11.9 K 2023-04-13 20:54:26,239 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/0f9e638280024f1d90839dd0a78cc8c2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2 2023-04-13 20:54:26,243 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2, entries=150, sequenceid=268, filesize=11.9 K 2023-04-13 20:54:26,244 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 12f4a42416bb74102917a94867a06dfb in 889ms, sequenceid=268, compaction requested=false 2023-04-13 20:54:26,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:26,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40882fde[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,251 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,254 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,254 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,258 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,259 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:26,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,262 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,262 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:26,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:26,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:26,362 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:26,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:26,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:26,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:26,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:26,368 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:26,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:26,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:26,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,371 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:26,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:26,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:26,374 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,374 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:26,374 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:26,374 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,376 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a9b43cb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,382 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419326382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419326383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,385 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,385 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,388 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,389 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:26,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,392 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,393 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,393 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419326486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419326486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,493 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:26,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:26,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:26,496 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,499 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:26,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:26,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:26,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:26,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:26,501 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,501 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:26,501 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:26,501 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4255ac0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,506 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,513 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,513 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,513 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,513 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,513 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,515 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,515 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,517 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,519 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,520 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,520 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,520 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,620 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:26,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:26,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:26,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:26,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:26,625 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:26,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:26,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:26,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,626 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:26,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:26,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:26,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:26,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:26,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:26,629 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37c63d28[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,634 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,634 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,634 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,635 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,635 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,636 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,638 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,638 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,639 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,640 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:26,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,644 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,645 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,645 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419326687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419326688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,745 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:26,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:26,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:26,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:26,748 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:26,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:26,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:26,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,750 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,751 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:26,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:26,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:26,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:26,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:26,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:26,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,754 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363102d1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,768 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,770 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133a15f2fdee5a4d58aa6e6e176d809238 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133a15f2fdee5a4d58aa6e6e176d809238 2023-04-13 20:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,771 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,771 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,773 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,778 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:26,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,779 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,779 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,879 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:26,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:26,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:26,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:26,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:26,883 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:26,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:26,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:26,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,885 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:26,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:26,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:26,886 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,886 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:26,886 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:26,887 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:26,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:26,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6848ac7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:26,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,892 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:26,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:26,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:26,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:26,897 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:26,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:26,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:26,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:26,897 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:26,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:26,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:26,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:26,905 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:26,906 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:26,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:26,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:26,906 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:26,906 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:26,906 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:26,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419326992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:26,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:26,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419326992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,006 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,009 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:27,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:27,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:27,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:27,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d9338d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:27,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:27,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:27,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:27,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:27,024 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:27,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:27,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:27,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:27,025 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:27,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,027 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:27,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:27,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:27,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,041 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,041 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:27,141 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,145 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:27,149 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:27,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:27,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:27,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e7064c3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:27,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,154 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:27,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:27,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:27,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:27,155 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:27,158 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:27,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:27,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:27,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:27,158 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:27,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:27,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,165 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:27,165 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:27,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,179 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/aaea1d024ff94b70a6a02d664d6ef617 2023-04-13 20:54:27,265 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,268 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:27,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:27,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:27,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1069dc56[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:27,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,276 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:27,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:27,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:27,279 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:27,282 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:27,282 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:27,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:27,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:27,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,291 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,291 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,291 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,291 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,291 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:27,291 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:27,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,391 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,394 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:27,403 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3219ae2e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:27,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:27,405 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:27,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:27,406 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,406 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:27,408 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:27,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:27,411 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:27,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:27,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:27,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:27,411 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:27,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:27,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,417 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:27,417 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:27,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:27,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419327494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:27,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419327497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,517 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,521 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:27,527 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53bdb6bb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:27,528 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:27,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:27,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:27,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:27,534 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:27,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:27,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:27,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:27,534 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:27,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:27,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:27,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:27,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:27,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:27,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:27,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:27,541 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:27,544 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:27,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/95455e6a12e645a29a9328d275d83cc4 2023-04-13 20:54:27,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/bcde282b340b41fdbbca64795722836c 2023-04-13 20:54:27,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/aaea1d024ff94b70a6a02d664d6ef617 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617 2023-04-13 20:54:27,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617, entries=250, sequenceid=285, filesize=39.2 K 2023-04-13 20:54:27,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/95455e6a12e645a29a9328d275d83cc4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4 2023-04-13 20:54:27,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4, entries=150, sequenceid=285, filesize=11.9 K 2023-04-13 20:54:27,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/bcde282b340b41fdbbca64795722836c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c 2023-04-13 20:54:27,644 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:27,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:27,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:27,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:27,649 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:27,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:27,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:27,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,650 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:27,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:27,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c, entries=150, sequenceid=285, filesize=11.9 K 2023-04-13 20:54:27,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:27,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:27,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:27,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:27,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 12f4a42416bb74102917a94867a06dfb in 1299ms, sequenceid=285, compaction requested=true 2023-04-13 20:54:27,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:27,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:27,653 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:27,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:27,653 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:27,653 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:27,654 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92602 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:27,654 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:27,654 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36938 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:27,654 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/5ea8f1f622b344518ddd41c35646acb0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=90.4 K 2023-04-13 20:54:27,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:27,655 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,655 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/f27a33496dc843e48a6dab0f738f069f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.1 K 2023-04-13 20:54:27,655 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5ea8f1f622b344518ddd41c35646acb0, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263575 2023-04-13 20:54:27,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f27a33496dc843e48a6dab0f738f069f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263575 2023-04-13 20:54:27,655 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7818d93af6504becb8441f539942b97c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681419263920 2023-04-13 20:54:27,656 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9a3cfb79db914c758c448019f51c4589, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681419263920 2023-04-13 20:54:27,656 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aaea1d024ff94b70a6a02d664d6ef617, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:27,656 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 95455e6a12e645a29a9328d275d83cc4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:27,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:27,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:27,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:27,668 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:27,669 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:27,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:27,677 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:27,686 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#182 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:27,710 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/642a3cd0a791403b873c8c2469fbc17b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/642a3cd0a791403b873c8c2469fbc17b 2023-04-13 20:54:27,716 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into 642a3cd0a791403b873c8c2469fbc17b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:27,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:27,716 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419267653; duration=0sec 2023-04-13 20:54:27,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:27,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:27,717 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36904 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:27,717 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:27,717 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:27,717 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cbaa9bf428c94f55b37a8ec4ede02adc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.0 K 2023-04-13 20:54:27,717 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cbaa9bf428c94f55b37a8ec4ede02adc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419263575 2023-04-13 20:54:27,718 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0f9e638280024f1d90839dd0a78cc8c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1681419263920 2023-04-13 20:54:27,719 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bcde282b340b41fdbbca64795722836c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:27,725 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#185 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:27,743 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/393d3fd5e59e49a59907b02d5abb0f4e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/393d3fd5e59e49a59907b02d5abb0f4e 2023-04-13 20:54:27,748 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into 393d3fd5e59e49a59907b02d5abb0f4e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:27,748 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:27,748 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=13, startTime=1681419267653; duration=0sec 2023-04-13 20:54:27,749 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:28,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,114 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b1920a7e6b9b48f7ba723b0f13de4819 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b1920a7e6b9b48f7ba723b0f13de4819 2023-04-13 20:54:28,120 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=301, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/2fcec33a73e448d9ab152e78b73a307e 2023-04-13 20:54:28,129 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/ad1f43800015435c8b5d92c8ce8988b4 2023-04-13 20:54:28,142 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/a5ecc8bcb11e4b80985d2f4d01275127 2023-04-13 20:54:28,146 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/2fcec33a73e448d9ab152e78b73a307e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e 2023-04-13 20:54:28,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419328149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,151 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e, entries=150, sequenceid=301, filesize=25.4 K 2023-04-13 20:54:28,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/ad1f43800015435c8b5d92c8ce8988b4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4 2023-04-13 20:54:28,159 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4, entries=150, sequenceid=301, filesize=11.9 K 2023-04-13 20:54:28,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/a5ecc8bcb11e4b80985d2f4d01275127 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127 2023-04-13 20:54:28,163 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127, entries=150, sequenceid=301, filesize=11.9 K 2023-04-13 20:54:28,164 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 12f4a42416bb74102917a94867a06dfb in 495ms, sequenceid=301, compaction requested=false 2023-04-13 20:54:28,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:28,164 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec68673[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,188 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,188 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:28,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:54:28,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:28,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:28,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:28,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:28,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:28,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:28,288 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,291 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419328292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,299 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20937e91[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,307 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,307 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,307 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,307 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,307 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,314 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,314 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419328395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,415 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,418 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,422 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,422 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,422 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,422 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,422 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f04cc73[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,431 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,432 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,437 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,437 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419328503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419328507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,526 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/cfc4eca96e9e49d295978097db3bdd7d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/cfc4eca96e9e49d295978097db3bdd7d 2023-04-13 20:54:28,530 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into cfc4eca96e9e49d295978097db3bdd7d(size=26.1 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:28,530 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:28,530 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419267653; duration=0sec 2023-04-13 20:54:28,530 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:28,537 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,540 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,544 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,544 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,544 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,545 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,546 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,546 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,546 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,548 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6132b146[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,561 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,561 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419328598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,661 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,664 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,670 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,671 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041309aa94c5c4b94d849fcb7bc465f7e717 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041309aa94c5c4b94d849fcb7bc465f7e717 2023-04-13 20:54:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b0d460d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,678 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,678 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,678 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,678 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,680 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,683 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,683 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,683 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,683 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,683 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,684 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,684 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,684 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,685 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,688 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,688 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,689 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:28,689 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,689 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,789 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,792 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,797 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,797 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,797 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,797 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10d1d20b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,803 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,805 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,805 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,808 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,809 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:28,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,812 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,812 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,812 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419328900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,912 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:28,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:28,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:28,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:28,916 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:28,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:28,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:28,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,917 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:28,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:28,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:28,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:28,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:28,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:28,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:28,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:28,921 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:28,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694ff336[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:28,923 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:28,923 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:28,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:28,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:28,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:28,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,926 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:28,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:28,930 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:28,930 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:28,930 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:28,930 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:28,930 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:28,931 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,931 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:28,931 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:28,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:28,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:28,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:28,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:28,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:28,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:28,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:28,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:28,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:28,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:28,936 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:28,936 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:28,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:28,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:28,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:28,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:28,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,036 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,039 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,048 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d83c7d8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,058 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,060 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,060 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,066 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,066 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,067 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,067 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,079 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/0efa582a3e10415ebd8cb983af550836 2023-04-13 20:54:29,167 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,170 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cffd74b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,183 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,185 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,186 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,194 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,194 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,194 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,294 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,298 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a01755[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,305 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,310 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,310 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,313 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,316 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,316 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419329401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,416 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,419 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,425 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,425 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54608a74[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,431 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,435 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,435 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,445 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,445 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,489 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/a97fa659b6a44dac882f030a4ddd800e 2023-04-13 20:54:29,545 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,548 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,550 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,551 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,551 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,551 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,552 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57de3a27[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,555 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,555 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,564 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,564 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,570 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,570 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,670 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,673 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,674 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,674 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,679 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7539fa18[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,682 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,682 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,686 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,686 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:29,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,693 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,693 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,793 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,796 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,800 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,800 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,800 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:29,800 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:29,800 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:29,802 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:29,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37614155[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:29,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:29,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,805 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:29,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,805 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:29,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,807 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,807 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,807 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:29,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:29,810 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:29,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:29,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:29,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:29,810 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:29,813 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:29,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:29,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:29,817 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:29,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:29,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:29,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:29,820 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:29,820 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:29,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6 2023-04-13 20:54:29,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/0efa582a3e10415ebd8cb983af550836 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836 2023-04-13 20:54:29,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836, entries=150, sequenceid=320, filesize=25.4 K 2023-04-13 20:54:29,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/a97fa659b6a44dac882f030a4ddd800e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e 2023-04-13 20:54:29,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e, entries=150, sequenceid=320, filesize=11.9 K 2023-04-13 20:54:29,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6 2023-04-13 20:54:29,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6, entries=150, sequenceid=320, filesize=11.9 K 2023-04-13 20:54:29,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 12f4a42416bb74102917a94867a06dfb in 1664ms, sequenceid=320, compaction requested=true 2023-04-13 20:54:29,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:29,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:29,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:29,919 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:29,920 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:29,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:29,920 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:29,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78820 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:29,921 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,921 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,921 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/cfc4eca96e9e49d295978097db3bdd7d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=77.0 K 2023-04-13 20:54:29,921 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/642a3cd0a791403b873c8c2469fbc17b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.4 K 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cfc4eca96e9e49d295978097db3bdd7d, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:29,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 642a3cd0a791403b873c8c2469fbc17b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:29,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2fcec33a73e448d9ab152e78b73a307e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681419266379 2023-04-13 20:54:29,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ad1f43800015435c8b5d92c8ce8988b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681419266379 2023-04-13 20:54:29,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0efa582a3e10415ebd8cb983af550836, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:29,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:29,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a97fa659b6a44dac882f030a4ddd800e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:29,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:29,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:29,924 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,927 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:29,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:29,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:29,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:29,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:29,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:29,936 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:29,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:29,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:29,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:29,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:29,948 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:29,948 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:29,950 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#191 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:30,363 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/d35321d1e13b406087ed6c38958a111e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d35321d1e13b406087ed6c38958a111e 2023-04-13 20:54:30,370 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into d35321d1e13b406087ed6c38958a111e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:30,370 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:30,371 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419269919; duration=0sec 2023-04-13 20:54:30,371 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:30,371 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:30,372 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:30,372 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:30,372 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:30,372 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/393d3fd5e59e49a59907b02d5abb0f4e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.3 K 2023-04-13 20:54:30,373 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 393d3fd5e59e49a59907b02d5abb0f4e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1681419265715 2023-04-13 20:54:30,373 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a5ecc8bcb11e4b80985d2f4d01275127, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1681419266379 2023-04-13 20:54:30,373 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 61ebf3fdaa3d4fd3b6a082e2fb9de2f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:30,379 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#194 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:30,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/5532886d3c1b4b6483a143f41bbc8fbb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5532886d3c1b4b6483a143f41bbc8fbb 2023-04-13 20:54:30,393 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413afcb28817a7c42aeb3034d7dbf7295d8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afcb28817a7c42aeb3034d7dbf7295d8 2023-04-13 20:54:30,396 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into 5532886d3c1b4b6483a143f41bbc8fbb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:30,397 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:30,397 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=13, startTime=1681419269920; duration=0sec 2023-04-13 20:54:30,397 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:30,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:30,414 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/7cf77df3b91c4da2ad8d6e2da1646157 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7cf77df3b91c4da2ad8d6e2da1646157 2023-04-13 20:54:30,419 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into 7cf77df3b91c4da2ad8d6e2da1646157(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:30,419 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:30,419 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419269919; duration=0sec 2023-04-13 20:54:30,419 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:30,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:30,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419330441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:30,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:30,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419330507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:30,510 DEBUG [Thread-925] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:30,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:30,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419330515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:30,517 DEBUG [Thread-923] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:30,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:30,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419330544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:30,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:30,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419330746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:30,808 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/04bb5ba4d82049dbaf05dbf36d4aa5cb 2023-04-13 20:54:31,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:31,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419331050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,219 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/4e5e9065829a44c58d33f42c2ba6ee20 2023-04-13 20:54:31,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:31,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419331552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,630 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/f42d543300eb47c8b9b1cdfc7b620295 2023-04-13 20:54:31,634 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/04bb5ba4d82049dbaf05dbf36d4aa5cb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb 2023-04-13 20:54:31,638 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb, entries=150, sequenceid=336, filesize=25.4 K 2023-04-13 20:54:31,639 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/4e5e9065829a44c58d33f42c2ba6ee20 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20 2023-04-13 20:54:31,643 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20, entries=150, sequenceid=336, filesize=11.9 K 2023-04-13 20:54:31,644 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/f42d543300eb47c8b9b1cdfc7b620295 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295 2023-04-13 20:54:31,647 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295, entries=150, sequenceid=336, filesize=11.9 K 2023-04-13 20:54:31,648 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 12f4a42416bb74102917a94867a06dfb in 1700ms, sequenceid=336, compaction requested=false 2023-04-13 20:54:31,648 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:31,648 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:31,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:31,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:31,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67024ee7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:31,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:31,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:31,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:31,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:31,658 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:31,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:31,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:31,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:31,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:31,661 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:31,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:31,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:31,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:31,661 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:31,664 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,664 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:31,664 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:31,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:31,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:31,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:31,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:31,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:31,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:31,668 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:31,668 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:31,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:31,768 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:31,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:31,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:31,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:31,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:31,772 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:31,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:31,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:31,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:31,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:31,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:31,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:31,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:31,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:31,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:31,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:31,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:31,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:31,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:31,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:31,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:31,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:31,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:31,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:31,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:31,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:31,776 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:31,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:31,788 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f09b36fe880e4ad19015551a4842e1fd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f09b36fe880e4ad19015551a4842e1fd 2023-04-13 20:54:32,194 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=356, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/4bd55a1655234690b42b55d55510806a 2023-04-13 20:54:32,206 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/e7cf6a80f71149b886d69690c1aaa402 2023-04-13 20:54:32,219 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/6bd7ed314f1f43b0856ee8cfe48ac4f9 2023-04-13 20:54:32,226 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/4bd55a1655234690b42b55d55510806a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a 2023-04-13 20:54:32,231 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a, entries=150, sequenceid=356, filesize=25.4 K 2023-04-13 20:54:32,232 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/e7cf6a80f71149b886d69690c1aaa402 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402 2023-04-13 20:54:32,236 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402, entries=150, sequenceid=356, filesize=11.9 K 2023-04-13 20:54:32,237 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/6bd7ed314f1f43b0856ee8cfe48ac4f9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9 2023-04-13 20:54:32,245 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9, entries=150, sequenceid=356, filesize=11.9 K 2023-04-13 20:54:32,246 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 12f4a42416bb74102917a94867a06dfb in 470ms, sequenceid=356, compaction requested=true 2023-04-13 20:54:32,246 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:32,246 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:32,248 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,248 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:32,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ab5f22[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:32,251 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:32,251 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,253 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:32,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:32,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:32,258 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:32,258 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:32,258 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:32,258 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:32,258 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:32,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,260 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,260 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:32,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:32,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,282 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:32,282 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,382 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:32,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:32,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:32,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:32,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:32,386 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:32,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:32,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:32,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,390 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:32,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:32,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:32,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:32,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:32,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:32,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:32,392 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:32,392 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,392 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:32,392 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:32,392 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:32,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:32,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:32,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:32,395 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,395 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:32,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:32,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b788435[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:32,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:32,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:32,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:32,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:32,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:32,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:32,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:32,402 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:32,402 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:32,402 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:32,402 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:32,402 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:32,405 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,405 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,405 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:32,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:32,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,411 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:32,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,411 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:32,511 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:32,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:32,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:32,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:32,516 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:32,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:32,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:32,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,516 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:32,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:32,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:32,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:32,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:32,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:32,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:32,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:32,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:32,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,525 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:32,527 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:32,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46ae09e6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:32,527 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:32,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:32,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:32,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:32,531 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:32,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:32,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:32,534 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:32,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:32,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:32,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:32,535 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:32,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:32,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:32,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:32,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:32,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:32,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:32,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:32,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,542 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:32,542 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:32,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:32,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:32,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,642 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:32,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:32,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:32,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:32,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:32,646 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:32,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:32,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:32,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:32,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:32,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:32,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:32,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:32,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:32,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:32,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:32,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:32,650 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:32,650 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:32,650 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:32,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:32,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:32,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:32,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:32,656 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 20:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:32,656 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:32,657 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:32,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:32,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419272705, max timestamp=1681419272724Num uniques 859; 2023-04-13 20:54:32,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:32,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419272705, max timestamp=1681419272724Num uniques 859; 2023-04-13 20:54:32,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:32,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419272705, max timestamp=1681419272724Num uniques 859; 2023-04-13 20:54:32,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:32,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419272747, max timestamp=1681419272784Num uniques 1118; 2023-04-13 20:54:32,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-13 20:54:32,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419272747, max timestamp=1681419272784Num uniques 1118; 2023-04-13 20:54:32,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-13 20:54:32,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419272747, max timestamp=1681419272784Num uniques 1118; 2023-04-13 20:54:32,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-04-13 20:54:32,864 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419272835, max timestamp=1681419272862Num uniques 1077; 2023-04-13 20:54:32,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419272835, max timestamp=1681419272862Num uniques 1077; 2023-04-13 20:54:32,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-13 20:54:32,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419272835, max timestamp=1681419272862Num uniques 1077; 2023-04-13 20:54:32,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-13 20:54:32,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419272911, max timestamp=1681419272927Num uniques 1086; 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419272911, max timestamp=1681419272927Num uniques 1086; 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419272911, max timestamp=1681419272927Num uniques 1086; 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:54:32,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419272963, max timestamp=1681419273000Num uniques 1045; 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419272963, max timestamp=1681419273000Num uniques 1045; 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419272963, max timestamp=1681419273000Num uniques 1045; 2023-04-13 20:54:33,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 20:54:33,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419273056, max timestamp=1681419273072Num uniques 1104; 2023-04-13 20:54:33,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419273056, max timestamp=1681419273072Num uniques 1104; 2023-04-13 20:54:33,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 20:54:33,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419273056, max timestamp=1681419273072Num uniques 1104; 2023-04-13 20:54:33,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 20:54:33,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-13 20:54:33,095 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041305af54382df542cabea8b8cda8f6d861 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305af54382df542cabea8b8cda8f6d861 2023-04-13 20:54:33,104 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=365, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/6d1a97b1e0ea43d5900fb09afd839507 2023-04-13 20:54:33,131 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a 2023-04-13 20:54:33,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419273119, max timestamp=1681419273147Num uniques 1063; 2023-04-13 20:54:33,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419273119, max timestamp=1681419273147Num uniques 1063; 2023-04-13 20:54:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 20:54:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 20:54:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419273119, max timestamp=1681419273147Num uniques 1063; 2023-04-13 20:54:33,150 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-13 20:54:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419273159, max timestamp=1681419273198Num uniques 1072; 2023-04-13 20:54:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419273159, max timestamp=1681419273198Num uniques 1072; 2023-04-13 20:54:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:33,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419273159, max timestamp=1681419273198Num uniques 1072; 2023-04-13 20:54:33,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681419273227, max timestamp=1681419273263Num uniques 1081; 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681419273227, max timestamp=1681419273263Num uniques 1081; 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1681419273227, max timestamp=1681419273263Num uniques 1081; 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-13 20:54:33,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-04-13 20:54:33,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419273304, max timestamp=1681419273318Num uniques 1140; 2023-04-13 20:54:33,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-13 20:54:33,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419273304, max timestamp=1681419273318Num uniques 1140; 2023-04-13 20:54:33,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-13 20:54:33,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419273304, max timestamp=1681419273318Num uniques 1140; 2023-04-13 20:54:33,323 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-04-13 20:54:33,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419273352, max timestamp=1681419273375Num uniques 1049; 2023-04-13 20:54:33,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419273352, max timestamp=1681419273375Num uniques 1049; 2023-04-13 20:54:33,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:54:33,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419273352, max timestamp=1681419273375Num uniques 1049; 2023-04-13 20:54:33,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:54:33,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681419273411, max timestamp=1681419273444Num uniques 1108; 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681419273411, max timestamp=1681419273444Num uniques 1108; 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1681419273411, max timestamp=1681419273444Num uniques 1108; 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-13 20:54:33,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-13 20:54:33,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681419273496, max timestamp=1681419273514Num uniques 1067; 2023-04-13 20:54:33,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 20:54:33,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681419273496, max timestamp=1681419273514Num uniques 1067; 2023-04-13 20:54:33,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 20:54:33,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1681419273496, max timestamp=1681419273514Num uniques 1067; 2023-04-13 20:54:33,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-13 20:54:33,546 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/6d6e4ce3689a4d3693cb8efbd5068765 2023-04-13 20:54:33,551 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/6d1a97b1e0ea43d5900fb09afd839507 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507 2023-04-13 20:54:33,555 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507, entries=200, sequenceid=365, filesize=32.3 K 2023-04-13 20:54:33,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a 2023-04-13 20:54:33,560 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a, entries=150, sequenceid=365, filesize=11.9 K 2023-04-13 20:54:33,561 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/6d6e4ce3689a4d3693cb8efbd5068765 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765 2023-04-13 20:54:33,565 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765, entries=150, sequenceid=365, filesize=11.9 K 2023-04-13 20:54:33,565 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=100.63 KB/103050 for 12f4a42416bb74102917a94867a06dfb in 909ms, sequenceid=365, compaction requested=true 2023-04-13 20:54:33,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:33,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:33,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:33,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:33,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:33,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:33,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:33,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@332e12a2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:33,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:33,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:33,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:33,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:33,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:33,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:33,575 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=107.34 KB heapSize=304.55 KB 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:33,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:33,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:33,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:33,577 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:33,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:33,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:33,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:33,578 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:33,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:33,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:33,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,586 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:33,586 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:33,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,622 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041395b31cd0f052460d884c9bc71c5cfbe2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041395b31cd0f052460d884c9bc71c5cfbe2 2023-04-13 20:54:33,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:33,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 455 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419333640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,686 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:33,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:33,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:33,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:33,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:33,690 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,691 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:33,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:33,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:33,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:33,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:33,693 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,693 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:33,693 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:33,693 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:33,695 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,696 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,696 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:33,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@286709aa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:33,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:33,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:33,702 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:33,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:33,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:33,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:33,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:33,706 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:33,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:33,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:33,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:33,706 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:33,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:33,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:33,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,713 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:33,713 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:33,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:33,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 457 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419333742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,813 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:33,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:33,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:33,816 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:33,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,819 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:33,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:33,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:33,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:33,821 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,821 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:33,821 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:33,821 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,821 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:33,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:33,823 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:33,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@178179bf[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:33,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:33,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:33,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:33,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:33,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:33,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:33,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:33,838 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:33,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:33,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:33,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:33,838 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:33,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:33,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:33,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:33,847 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:33,847 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:33,947 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:33,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:33,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:33,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:33,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:33,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:33,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419333946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,950 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:33,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:33,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:33,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,952 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:33,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:33,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:33,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:33,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:33,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:33,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:33,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,965 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,966 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,966 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:33,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d6b390[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:33,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:33,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:33,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:33,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:33,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:33,972 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:33,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:33,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:33,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:33,973 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:33,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:33,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:33,975 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:33,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:33,980 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:33,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:33,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:33,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:33,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:33,981 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:33,981 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,981 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:33,981 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:33,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:33,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:33,981 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:33,981 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:34,035 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=604, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/aedc012aa8f24227bc881556a4daf922 2023-04-13 20:54:34,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/e84866c07bbe412f85c4ae1cef2676b9 2023-04-13 20:54:34,081 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:34,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/e5da21e4dbca4a1a990f848d47fc0d0f 2023-04-13 20:54:34,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:34,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:34,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:34,086 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:34,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:34,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:34,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:34,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/aedc012aa8f24227bc881556a4daf922 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922 2023-04-13 20:54:34,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:34,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,089 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:34,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:34,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:34,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:34,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:34,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:34,092 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,093 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:34,093 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:34,093 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:34,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:34,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922, entries=200, sequenceid=604, filesize=32.3 K 2023-04-13 20:54:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:34,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:34,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622e0e0a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:34,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:34,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,099 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/e84866c07bbe412f85c4ae1cef2676b9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9 2023-04-13 20:54:34,101 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:34,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:34,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:34,101 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,102 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:34,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:34,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:34,106 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9, entries=150, sequenceid=604, filesize=11.9 K 2023-04-13 20:54:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,107 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/e5da21e4dbca4a1a990f848d47fc0d0f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f 2023-04-13 20:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:34,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:34,109 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:34,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:34,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:34,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:34,109 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:34,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f, entries=150, sequenceid=604, filesize=11.9 K 2023-04-13 20:54:34,112 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,112 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:34,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,112 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~304.50 KB/311808, currentSize=73.80 KB/75570 for 12f4a42416bb74102917a94867a06dfb in 538ms, sequenceid=604, compaction requested=true 2023-04-13 20:54:34,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:34,112 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,113 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:34,112 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:34,113 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:34,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:34,113 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:34,114 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:34,115 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145094 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:34,115 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:34,115 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,115 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7cf77df3b91c4da2ad8d6e2da1646157, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=141.7 K 2023-04-13 20:54:34,116 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,116 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:34,116 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,116 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d35321d1e13b406087ed6c38958a111e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=60.3 K 2023-04-13 20:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:34,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,116 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7cf77df3b91c4da2ad8d6e2da1646157, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:34,117 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d35321d1e13b406087ed6c38958a111e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,117 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4e5e9065829a44c58d33f42c2ba6ee20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681419268277 2023-04-13 20:54:34,117 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 04bb5ba4d82049dbaf05dbf36d4aa5cb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681419268277 2023-04-13 20:54:34,118 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e7cf6a80f71149b886d69690c1aaa402, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681419270432 2023-04-13 20:54:34,118 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 4bd55a1655234690b42b55d55510806a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681419270432 2023-04-13 20:54:34,118 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6dd0d3ff51a34a60a1eeacfcd29a6b7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419272591 2023-04-13 20:54:34,119 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e84866c07bbe412f85c4ae1cef2676b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:34,119 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6d1a97b1e0ea43d5900fb09afd839507, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419272591 2023-04-13 20:54:34,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:34,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:34,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:34,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:34,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:34,120 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:34,120 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:34,120 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aedc012aa8f24227bc881556a4daf922, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:34,139 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#207 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:34,155 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#206 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:34,186 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/e3b8be46e734422eb7674c464548a183 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e3b8be46e734422eb7674c464548a183 2023-04-13 20:54:34,192 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into e3b8be46e734422eb7674c464548a183(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:34,192 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:34,193 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=11, startTime=1681419274113; duration=0sec 2023-04-13 20:54:34,193 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:34,193 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:34,197 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:34,197 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:34,197 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,197 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5532886d3c1b4b6483a143f41bbc8fbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=60.2 K 2023-04-13 20:54:34,198 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5532886d3c1b4b6483a143f41bbc8fbb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1681419268124 2023-04-13 20:54:34,198 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f42d543300eb47c8b9b1cdfc7b620295, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1681419268277 2023-04-13 20:54:34,199 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6bd7ed314f1f43b0856ee8cfe48ac4f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1681419270432 2023-04-13 20:54:34,199 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6d6e4ce3689a4d3693cb8efbd5068765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419272591 2023-04-13 20:54:34,199 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e5da21e4dbca4a1a990f848d47fc0d0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:34,220 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:34,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:34,222 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#208 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:34,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:34,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:34,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:34,225 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:34,225 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:34,225 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:34,226 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:34,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:34,228 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:34,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:34,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:34,230 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:34,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:34,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:34,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:34,248 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:34,248 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. started... 2023-04-13 20:54:34,249 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:34,249 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:34,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:34,257 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. as already flushing 2023-04-13 20:54:34,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419334313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419334417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47224 deadline: 1681419334533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,537 DEBUG [Thread-925] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:34,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47162 deadline: 1681419334555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,559 DEBUG [Thread-923] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8176 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:54:34,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 480 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419334622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:34,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/e7193f24aa16446ebad25ca1bb8dcae8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e7193f24aa16446ebad25ca1bb8dcae8 2023-04-13 20:54:34,659 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into e7193f24aa16446ebad25ca1bb8dcae8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:34,659 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:34,659 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=11, startTime=1681419274113; duration=0sec 2023-04-13 20:54:34,659 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:34,663 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/ed6861522bc945499d65c0b2c962eff7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/ed6861522bc945499d65c0b2c962eff7 2023-04-13 20:54:34,667 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into ed6861522bc945499d65c0b2c962eff7(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:34,667 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:34,667 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=11, startTime=1681419274112; duration=0sec 2023-04-13 20:54:34,667 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:34,681 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413872f4e54497c4d4d899840e65359f75b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872f4e54497c4d4d899840e65359f75b 2023-04-13 20:54:34,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:34,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419334926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,017 DEBUG [Thread-930] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x508246cc to 127.0.0.1:54584 2023-04-13 20:54:35,017 DEBUG [Thread-930] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:35,018 DEBUG [Thread-932] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08795451 to 127.0.0.1:54584 2023-04-13 20:54:35,018 DEBUG [Thread-932] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:35,019 DEBUG [Thread-934] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58e27a32 to 127.0.0.1:54584 2023-04-13 20:54:35,019 DEBUG [Thread-934] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:35,023 DEBUG [Thread-936] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x092bed09 to 127.0.0.1:54584 2023-04-13 20:54:35,024 DEBUG [Thread-936] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:35,086 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=619, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/a0ee7fb084f74c69857809ce4f1d1151 2023-04-13 20:54:35,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:35,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47208 deadline: 1681419335430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,495 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/d1af6185b87844dba9765f85efff6094 2023-04-13 20:54:35,904 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=619 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/1f58409cd34f435eb2367325d3d76f02 2023-04-13 20:54:35,908 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/a0ee7fb084f74c69857809ce4f1d1151 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151 2023-04-13 20:54:35,911 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151, entries=150, sequenceid=619, filesize=25.4 K 2023-04-13 20:54:35,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/d1af6185b87844dba9765f85efff6094 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094 2023-04-13 20:54:35,915 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094, entries=150, sequenceid=619, filesize=11.9 K 2023-04-13 20:54:35,915 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/1f58409cd34f435eb2367325d3d76f02 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02 2023-04-13 20:54:35,918 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02, entries=150, sequenceid=619, filesize=11.9 K 2023-04-13 20:54:35,919 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 12f4a42416bb74102917a94867a06dfb in 1670ms, sequenceid=619, compaction requested=false 2023-04-13 20:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:35,919 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:35,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:35,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:35,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:35,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:35,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:35,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:35,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@721471bc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:35,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:35,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:35,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:35,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:35,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:35,930 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:35,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:35,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:35,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:35,932 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:35,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:35,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:35,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:35,932 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:35,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:35,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:35,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:35,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:35,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:35,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:35,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:35,937 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:35,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:35,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:35,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:35,937 DEBUG [Thread-929] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:35,937 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:35,937 DEBUG [Thread-929] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:35,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:35,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:36,037 DEBUG [Thread-929] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:36,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:36,139 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:54:36,139 DEBUG [Thread-919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33a47c05 to 127.0.0.1:54584 2023-04-13 20:54:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:36,139 DEBUG [Thread-919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:36,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:36,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:36,140 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:36,140 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:36,144 DEBUG [Thread-921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21265fcb to 127.0.0.1:54584 2023-04-13 20:54:36,144 DEBUG [Thread-921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:36,432 DEBUG [Thread-927] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54451f11 to 127.0.0.1:54584 2023-04-13 20:54:36,432 DEBUG [Thread-927] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:36,551 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413df4394b011014185a1814a31e461e53f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df4394b011014185a1814a31e461e53f 2023-04-13 20:54:36,956 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=640, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/6dffd55d2818456a8f1c4a36e6a3e88d 2023-04-13 20:54:37,365 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/497cd64fb1794578b6d5b033817bd02a 2023-04-13 20:54:37,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/860d9edef9e443b2b6f028a506e50664 2023-04-13 20:54:37,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/6dffd55d2818456a8f1c4a36e6a3e88d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d 2023-04-13 20:54:37,781 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d, entries=150, sequenceid=640, filesize=25.4 K 2023-04-13 20:54:37,782 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/497cd64fb1794578b6d5b033817bd02a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a 2023-04-13 20:54:37,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a, entries=150, sequenceid=640, filesize=11.9 K 2023-04-13 20:54:37,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/860d9edef9e443b2b6f028a506e50664 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664 2023-04-13 20:54:37,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664, entries=150, sequenceid=640, filesize=11.9 K 2023-04-13 20:54:37,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for 12f4a42416bb74102917a94867a06dfb in 1650ms, sequenceid=640, compaction requested=true 2023-04-13 20:54:37,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:37,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:37,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:37,790 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:37,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:37,790 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79092 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/A is initiating minor compaction (all files) 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/B is initiating minor compaction (all files) 2023-04-13 20:54:37,791 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/A in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:37,791 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/B in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:37,791 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/ed6861522bc945499d65c0b2c962eff7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=77.2 K 2023-04-13 20:54:37,791 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e3b8be46e734422eb7674c464548a183, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.6 K 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ed6861522bc945499d65c0b2c962eff7, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:37,791 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e3b8be46e734422eb7674c464548a183, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:37,792 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a0ee7fb084f74c69857809ce4f1d1151, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681419273619 2023-04-13 20:54:37,792 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d1af6185b87844dba9765f85efff6094, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681419273619 2023-04-13 20:54:37,792 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6dffd55d2818456a8f1c4a36e6a3e88d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681419274294 2023-04-13 20:54:37,792 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 497cd64fb1794578b6d5b033817bd02a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681419274294 2023-04-13 20:54:37,799 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#B#compaction#215 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:37,805 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#A#compaction#216 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:38,210 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/6eaf53207c0b45a685792af8ad864db7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6eaf53207c0b45a685792af8ad864db7 2023-04-13 20:54:38,214 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/B of 12f4a42416bb74102917a94867a06dfb into 6eaf53207c0b45a685792af8ad864db7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:38,215 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:38,215 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/B, priority=13, startTime=1681419277790; duration=0sec 2023-04-13 20:54:38,215 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:38,215 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:38,215 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:38,216 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 12f4a42416bb74102917a94867a06dfb/C is initiating minor compaction (all files) 2023-04-13 20:54:38,216 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 12f4a42416bb74102917a94867a06dfb/C in TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:38,216 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e7193f24aa16446ebad25ca1bb8dcae8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp, totalSize=36.6 K 2023-04-13 20:54:38,216 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e7193f24aa16446ebad25ca1bb8dcae8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1681419273519 2023-04-13 20:54:38,216 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1f58409cd34f435eb2367325d3d76f02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=619, earliestPutTs=1681419273619 2023-04-13 20:54:38,217 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 860d9edef9e443b2b6f028a506e50664, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1681419274294 2023-04-13 20:54:38,223 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 12f4a42416bb74102917a94867a06dfb#C#compaction#217 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:38,630 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/cce80b72f9534af9be52106d875e6805 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cce80b72f9534af9be52106d875e6805 2023-04-13 20:54:38,634 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/C of 12f4a42416bb74102917a94867a06dfb into cce80b72f9534af9be52106d875e6805(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:38,634 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:38,634 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/C, priority=13, startTime=1681419277790; duration=0sec 2023-04-13 20:54:38,634 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:39,021 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/780aef041cce45eb80ffe808a437a9a6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/780aef041cce45eb80ffe808a437a9a6 2023-04-13 20:54:39,025 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 12f4a42416bb74102917a94867a06dfb/A of 12f4a42416bb74102917a94867a06dfb into 780aef041cce45eb80ffe808a437a9a6(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 20:54:39,025 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:39,025 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb., storeName=12f4a42416bb74102917a94867a06dfb/A, priority=13, startTime=1681419277790; duration=1sec 2023-04-13 20:54:39,025 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:39,177 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 20:54:44,586 DEBUG [Thread-925] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0feb770d to 127.0.0.1:54584 2023-04-13 20:54:44,586 DEBUG [Thread-925] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:44,615 DEBUG [Thread-923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69f15cb4 to 127.0.0.1:54584 2023-04-13 20:54:44,615 DEBUG [Thread-923] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 65 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 54 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 373 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:54:44,615 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4846 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4833 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 2083 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 6248 rows 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 2093 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 6278 rows 2023-04-13 20:54:44,616 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:54:44,616 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0186ae08 to 127.0.0.1:54584 2023-04-13 20:54:44,616 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:54:44,620 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:54:44,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:54:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:44,625 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419284625"}]},"ts":"1681419284625"} 2023-04-13 20:54:44,626 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:54:44,628 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:54:44,629 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, UNASSIGN}] 2023-04-13 20:54:44,630 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, UNASSIGN 2023-04-13 20:54:44,630 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:44,630 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419284630"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419284630"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419284630"}]},"ts":"1681419284630"} 2023-04-13 20:54:44,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:44,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:44,783 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 12f4a42416bb74102917a94867a06dfb, disabling compactions & flushes 2023-04-13 20:54:44,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. after waiting 0 ms 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:44,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 12f4a42416bb74102917a94867a06dfb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=A 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=B 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 12f4a42416bb74102917a94867a06dfb, store=C 2023-04-13 20:54:44,784 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:44,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:45,197 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413bd6d54919d6d4f06ba06107d56e5b2a9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413bd6d54919d6d4f06ba06107d56e5b2a9 2023-04-13 20:54:45,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:45,601 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=650, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/725a66c90c484819844c07f026b98d75 2023-04-13 20:54:45,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=650 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/8c494e2ee1cd4194b451b744a61b530e 2023-04-13 20:54:45,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:46,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=650 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/500c9fbaf28e4a90a22ff7186d423918 2023-04-13 20:54:46,026 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/A/725a66c90c484819844c07f026b98d75 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/725a66c90c484819844c07f026b98d75 2023-04-13 20:54:46,029 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/725a66c90c484819844c07f026b98d75, entries=100, sequenceid=650, filesize=18.6 K 2023-04-13 20:54:46,029 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/B/8c494e2ee1cd4194b451b744a61b530e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c494e2ee1cd4194b451b744a61b530e 2023-04-13 20:54:46,033 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c494e2ee1cd4194b451b744a61b530e, entries=100, sequenceid=650, filesize=9.5 K 2023-04-13 20:54:46,033 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/.tmp/C/500c9fbaf28e4a90a22ff7186d423918 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/500c9fbaf28e4a90a22ff7186d423918 2023-04-13 20:54:46,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/500c9fbaf28e4a90a22ff7186d423918, entries=100, sequenceid=650, filesize=9.5 K 2023-04-13 20:54:46,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 12f4a42416bb74102917a94867a06dfb in 1253ms, sequenceid=650, compaction requested=false 2023-04-13 20:54:46,058 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/336caaf9e491429f9e1e15a775825b8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/70cc38487fcd42ccaeee1e8232aabd04, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/acb84ec90c3041bb982552b9a1a00513, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/da7bf5c8313042d592fbbf6216eeb814, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/5ea8f1f622b344518ddd41c35646acb0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/cfc4eca96e9e49d295978097db3bdd7d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7cf77df3b91c4da2ad8d6e2da1646157, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/ed6861522bc945499d65c0b2c962eff7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d] to archive 2023-04-13 20:54:46,059 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:46,061 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d6473826a1e244cf8593d164ea3f8a49 2023-04-13 20:54:46,062 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9fc16ef337124c00ad19f89d027ddcd9 2023-04-13 20:54:46,063 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0edde9b536f1493a9a33da8716ed05f4 2023-04-13 20:54:46,064 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/336caaf9e491429f9e1e15a775825b8d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/336caaf9e491429f9e1e15a775825b8d 2023-04-13 20:54:46,065 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/d667c7cbbccb4e6ea6594f8ce1d21d09 2023-04-13 20:54:46,066 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/441183c1dab9417faee20903acd94ed7 2023-04-13 20:54:46,067 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/36b7f4153b524becb3969f27fbc046bd 2023-04-13 20:54:46,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/53213e4478f74be3969f1e92e11efb54 2023-04-13 20:54:46,068 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/70cc38487fcd42ccaeee1e8232aabd04 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/70cc38487fcd42ccaeee1e8232aabd04 2023-04-13 20:54:46,069 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/b1a7b2063bfc426cb7abd98d6bb424ea 2023-04-13 20:54:46,070 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/3f848fe1514b434ca15243373385576e 2023-04-13 20:54:46,071 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/acb84ec90c3041bb982552b9a1a00513 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/acb84ec90c3041bb982552b9a1a00513 2023-04-13 20:54:46,072 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/9abe58a978d74b0da42d4969ec974c97 2023-04-13 20:54:46,073 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/799bf261e8ce4f8a9a9bef600d4a17df 2023-04-13 20:54:46,074 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/da7bf5c8313042d592fbbf6216eeb814 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/da7bf5c8313042d592fbbf6216eeb814 2023-04-13 20:54:46,075 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/fbb9592466e04b259b0d4cbdd62e2106 2023-04-13 20:54:46,076 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/11b0a94ce9c44457aabc8c8520abc778 2023-04-13 20:54:46,077 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/5ea8f1f622b344518ddd41c35646acb0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/5ea8f1f622b344518ddd41c35646acb0 2023-04-13 20:54:46,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7818d93af6504becb8441f539942b97c 2023-04-13 20:54:46,078 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aaea1d024ff94b70a6a02d664d6ef617 2023-04-13 20:54:46,079 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/cfc4eca96e9e49d295978097db3bdd7d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/cfc4eca96e9e49d295978097db3bdd7d 2023-04-13 20:54:46,081 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/2fcec33a73e448d9ab152e78b73a307e 2023-04-13 20:54:46,082 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7cf77df3b91c4da2ad8d6e2da1646157 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/7cf77df3b91c4da2ad8d6e2da1646157 2023-04-13 20:54:46,083 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/0efa582a3e10415ebd8cb983af550836 2023-04-13 20:54:46,084 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/04bb5ba4d82049dbaf05dbf36d4aa5cb 2023-04-13 20:54:46,085 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/4bd55a1655234690b42b55d55510806a 2023-04-13 20:54:46,086 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6d1a97b1e0ea43d5900fb09afd839507 2023-04-13 20:54:46,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/aedc012aa8f24227bc881556a4daf922 2023-04-13 20:54:46,087 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/ed6861522bc945499d65c0b2c962eff7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/ed6861522bc945499d65c0b2c962eff7 2023-04-13 20:54:46,088 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/a0ee7fb084f74c69857809ce4f1d1151 2023-04-13 20:54:46,089 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/6dffd55d2818456a8f1c4a36e6a3e88d 2023-04-13 20:54:46,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/c606210c3163433fbf98eb45bd08b7a3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ed77dd54c39e42238f49b7ccab77e60b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1464d01b07304f829133770470e464ed, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b18bac294abb4ea59568c9142d092755, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/f27a33496dc843e48a6dab0f738f069f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/642a3cd0a791403b873c8c2469fbc17b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d35321d1e13b406087ed6c38958a111e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e3b8be46e734422eb7674c464548a183, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a] to archive 2023-04-13 20:54:46,092 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:46,093 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1e10ba8a585941b9a5d5409101c239b6 2023-04-13 20:54:46,094 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b86efdc127e54778b02ddebea854a149 2023-04-13 20:54:46,095 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/c606210c3163433fbf98eb45bd08b7a3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/c606210c3163433fbf98eb45bd08b7a3 2023-04-13 20:54:46,096 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17a342b9e41b4d91829364fa5cff299e 2023-04-13 20:54:46,097 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c1036d64584489e89fea5f53ed914ff 2023-04-13 20:54:46,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/eda7af6a1dc441bb9cce17cb4e32ede5 2023-04-13 20:54:46,098 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4c9bb471dabe44e887dcb3416ab52554 2023-04-13 20:54:46,099 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ed77dd54c39e42238f49b7ccab77e60b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ed77dd54c39e42238f49b7ccab77e60b 2023-04-13 20:54:46,100 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/17829bbf330d4d438682e5b599f557b6 2023-04-13 20:54:46,101 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/96fc3365a55a402dba974bdb91f45fcb 2023-04-13 20:54:46,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1464d01b07304f829133770470e464ed to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/1464d01b07304f829133770470e464ed 2023-04-13 20:54:46,102 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/2ca4ee9347c6439db2a32112a422a2ad 2023-04-13 20:54:46,103 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a38d32f999854e1c90d9e8ec993d25de 2023-04-13 20:54:46,104 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b18bac294abb4ea59568c9142d092755 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/b18bac294abb4ea59568c9142d092755 2023-04-13 20:54:46,105 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6e89ea656c014e29b5394986aad86f1f 2023-04-13 20:54:46,106 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/11564e03580c418197ef44bc79209e56 2023-04-13 20:54:46,107 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/f27a33496dc843e48a6dab0f738f069f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/f27a33496dc843e48a6dab0f738f069f 2023-04-13 20:54:46,108 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/376bde5929254e1cb1d0466b1c1ff0b3 2023-04-13 20:54:46,109 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/9a3cfb79db914c758c448019f51c4589 2023-04-13 20:54:46,110 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/642a3cd0a791403b873c8c2469fbc17b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/642a3cd0a791403b873c8c2469fbc17b 2023-04-13 20:54:46,111 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/95455e6a12e645a29a9328d275d83cc4 2023-04-13 20:54:46,112 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/ad1f43800015435c8b5d92c8ce8988b4 2023-04-13 20:54:46,113 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d35321d1e13b406087ed6c38958a111e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d35321d1e13b406087ed6c38958a111e 2023-04-13 20:54:46,114 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/a97fa659b6a44dac882f030a4ddd800e 2023-04-13 20:54:46,115 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/4e5e9065829a44c58d33f42c2ba6ee20 2023-04-13 20:54:46,116 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e7cf6a80f71149b886d69690c1aaa402 2023-04-13 20:54:46,117 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6dd0d3ff51a34a60a1eeacfcd29a6b7a 2023-04-13 20:54:46,118 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e3b8be46e734422eb7674c464548a183 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e3b8be46e734422eb7674c464548a183 2023-04-13 20:54:46,119 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/e84866c07bbe412f85c4ae1cef2676b9 2023-04-13 20:54:46,120 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/d1af6185b87844dba9765f85efff6094 2023-04-13 20:54:46,121 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/497cd64fb1794578b6d5b033817bd02a 2023-04-13 20:54:46,123 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3b02e347c2248108839dbf48699afbe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/600361f72c8e473888abae393209b760, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cbaa9bf428c94f55b37a8ec4ede02adc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/393d3fd5e59e49a59907b02d5abb0f4e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5532886d3c1b4b6483a143f41bbc8fbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e7193f24aa16446ebad25ca1bb8dcae8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664] to archive 2023-04-13 20:54:46,124 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:54:46,125 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/af6ca7b777044efe939adc958edb595b 2023-04-13 20:54:46,126 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/83f8f6a623fb4c989d55ce11d9973dad 2023-04-13 20:54:46,127 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/187c4f0f6e9f4b3b9bfd26dc1b875c2e 2023-04-13 20:54:46,128 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/42c6bab0f6e945c88678af0f9c8e42a3 2023-04-13 20:54:46,129 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/992c8d3abb134bec91bb0589585d1906 2023-04-13 20:54:46,130 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/65159487ae5241798be2594e40bb52ac 2023-04-13 20:54:46,131 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3690bf020d244a6881a430175820e21 2023-04-13 20:54:46,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/7da124cff724409183bc11aec38cfa02 2023-04-13 20:54:46,132 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3b02e347c2248108839dbf48699afbe to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a3b02e347c2248108839dbf48699afbe 2023-04-13 20:54:46,133 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/14d0751ae4534967a7633cf0357ce3e2 2023-04-13 20:54:46,134 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/ffd4b621af714e529f37065cb2739149 2023-04-13 20:54:46,135 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/600361f72c8e473888abae393209b760 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/600361f72c8e473888abae393209b760 2023-04-13 20:54:46,136 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1ded7a1a45f34d639e31aec26252d2d2 2023-04-13 20:54:46,137 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/d7d93b46bc0742658d09e179f468a2a8 2023-04-13 20:54:46,138 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5646ac41683647c28524abc5fe5bc34b 2023-04-13 20:54:46,139 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cbaa9bf428c94f55b37a8ec4ede02adc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cbaa9bf428c94f55b37a8ec4ede02adc 2023-04-13 20:54:46,140 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/97fd8647ab444827a3dd778fb7f9ce5f 2023-04-13 20:54:46,141 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/0f9e638280024f1d90839dd0a78cc8c2 2023-04-13 20:54:46,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/393d3fd5e59e49a59907b02d5abb0f4e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/393d3fd5e59e49a59907b02d5abb0f4e 2023-04-13 20:54:46,142 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/bcde282b340b41fdbbca64795722836c 2023-04-13 20:54:46,143 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/a5ecc8bcb11e4b80985d2f4d01275127 2023-04-13 20:54:46,144 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5532886d3c1b4b6483a143f41bbc8fbb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/5532886d3c1b4b6483a143f41bbc8fbb 2023-04-13 20:54:46,145 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/61ebf3fdaa3d4fd3b6a082e2fb9de2f6 2023-04-13 20:54:46,146 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/f42d543300eb47c8b9b1cdfc7b620295 2023-04-13 20:54:46,147 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6bd7ed314f1f43b0856ee8cfe48ac4f9 2023-04-13 20:54:46,148 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/6d6e4ce3689a4d3693cb8efbd5068765 2023-04-13 20:54:46,149 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e7193f24aa16446ebad25ca1bb8dcae8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e7193f24aa16446ebad25ca1bb8dcae8 2023-04-13 20:54:46,150 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/e5da21e4dbca4a1a990f848d47fc0d0f 2023-04-13 20:54:46,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/1f58409cd34f435eb2367325d3d76f02 2023-04-13 20:54:46,151 DEBUG [StoreCloser-TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/860d9edef9e443b2b6f028a506e50664 2023-04-13 20:54:46,155 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits/653.seqid, newMaxSeqId=653, maxSeqId=4 2023-04-13 20:54:46,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb. 2023-04-13 20:54:46,157 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 12f4a42416bb74102917a94867a06dfb: 2023-04-13 20:54:46,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:46,159 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=12f4a42416bb74102917a94867a06dfb, regionState=CLOSED 2023-04-13 20:54:46,159 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419286159"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419286159"}]},"ts":"1681419286159"} 2023-04-13 20:54:46,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-13 20:54:46,161 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 12f4a42416bb74102917a94867a06dfb, server=jenkins-hbase4.apache.org,36431,1681419217857 in 1.5290 sec 2023-04-13 20:54:46,162 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-13 20:54:46,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=12f4a42416bb74102917a94867a06dfb, UNASSIGN in 1.5320 sec 2023-04-13 20:54:46,163 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419286163"}]},"ts":"1681419286163"} 2023-04-13 20:54:46,164 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:54:46,165 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:54:46,167 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5450 sec 2023-04-13 20:54:46,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-13 20:54:46,728 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-13 20:54:46,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:54:46,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,731 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 20:54:46,732 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,736 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:46,738 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits] 2023-04-13 20:54:46,740 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/725a66c90c484819844c07f026b98d75 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/725a66c90c484819844c07f026b98d75 2023-04-13 20:54:46,741 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/780aef041cce45eb80ffe808a437a9a6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/A/780aef041cce45eb80ffe808a437a9a6 2023-04-13 20:54:46,743 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6eaf53207c0b45a685792af8ad864db7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/6eaf53207c0b45a685792af8ad864db7 2023-04-13 20:54:46,745 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c494e2ee1cd4194b451b744a61b530e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/B/8c494e2ee1cd4194b451b744a61b530e 2023-04-13 20:54:46,747 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/500c9fbaf28e4a90a22ff7186d423918 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/500c9fbaf28e4a90a22ff7186d423918 2023-04-13 20:54:46,748 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cce80b72f9534af9be52106d875e6805 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/C/cce80b72f9534af9be52106d875e6805 2023-04-13 20:54:46,750 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits/653.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb/recovered.edits/653.seqid 2023-04-13 20:54:46,750 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/12f4a42416bb74102917a94867a06dfb 2023-04-13 20:54:46,751 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:54:46,751 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:54:46,751 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 20:54:46,754 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305af54382df542cabea8b8cda8f6d861 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041305af54382df542cabea8b8cda8f6d861 2023-04-13 20:54:46,755 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041309aa94c5c4b94d849fcb7bc465f7e717 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041309aa94c5c4b94d849fcb7bc465f7e717 2023-04-13 20:54:46,756 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041317729bd7dbb44e7eaa061e0a0b374317 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041317729bd7dbb44e7eaa061e0a0b374317 2023-04-13 20:54:46,758 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f6c37a473d4b5889340a07e92f4415 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f6c37a473d4b5889340a07e92f4415 2023-04-13 20:54:46,759 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133a15f2fdee5a4d58aa6e6e176d809238 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133a15f2fdee5a4d58aa6e6e176d809238 2023-04-13 20:54:46,760 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041355fd9bef4bf04d9abbeb757f9b80bbd1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041355fd9bef4bf04d9abbeb757f9b80bbd1 2023-04-13 20:54:46,762 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041371a3ce88feec438095b789d58739b6ba to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041371a3ce88feec438095b789d58739b6ba 2023-04-13 20:54:46,763 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413736ffd3fef8d4cef9147e4de6a7b50da to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413736ffd3fef8d4cef9147e4de6a7b50da 2023-04-13 20:54:46,764 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872e67f086064b9ab07f61ac6adcf948 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872e67f086064b9ab07f61ac6adcf948 2023-04-13 20:54:46,765 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872f4e54497c4d4d899840e65359f75b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413872f4e54497c4d4d899840e65359f75b 2023-04-13 20:54:46,766 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041395b31cd0f052460d884c9bc71c5cfbe2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041395b31cd0f052460d884c9bc71c5cfbe2 2023-04-13 20:54:46,767 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a9ef413f662c4a968a93ab15d08f843e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413a9ef413f662c4a968a93ab15d08f843e 2023-04-13 20:54:46,768 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afcb28817a7c42aeb3034d7dbf7295d8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413afcb28817a7c42aeb3034d7dbf7295d8 2023-04-13 20:54:46,769 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b1920a7e6b9b48f7ba723b0f13de4819 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b1920a7e6b9b48f7ba723b0f13de4819 2023-04-13 20:54:46,770 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b863d475ad9c49aa842dc546a62b18f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b863d475ad9c49aa842dc546a62b18f4 2023-04-13 20:54:46,771 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413bd6d54919d6d4f06ba06107d56e5b2a9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413bd6d54919d6d4f06ba06107d56e5b2a9 2023-04-13 20:54:46,772 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c26794c9d4ab4a52b567fc86bcfd02c9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c26794c9d4ab4a52b567fc86bcfd02c9 2023-04-13 20:54:46,773 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c2eac6531b514f11817542da9a57ac4c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c2eac6531b514f11817542da9a57ac4c 2023-04-13 20:54:46,774 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c7b2d4313e894968a9a9ce78ea6b028c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c7b2d4313e894968a9a9ce78ea6b028c 2023-04-13 20:54:46,775 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c9e471bb762244778cb3fb8b86dec3ef to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413c9e471bb762244778cb3fb8b86dec3ef 2023-04-13 20:54:46,776 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413dee8c190d1324cd4a9f2b4e6c37bf7c1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413dee8c190d1324cd4a9f2b4e6c37bf7c1 2023-04-13 20:54:46,777 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df4394b011014185a1814a31e461e53f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413df4394b011014185a1814a31e461e53f 2023-04-13 20:54:46,778 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f09b36fe880e4ad19015551a4842e1fd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f09b36fe880e4ad19015551a4842e1fd 2023-04-13 20:54:46,780 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5457c8c554b470db6780c118b6b8817 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f5457c8c554b470db6780c118b6b8817 2023-04-13 20:54:46,780 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:54:46,782 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,785 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:54:46,787 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:54:46,788 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,788 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:54:46,788 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419286788"}]},"ts":"9223372036854775807"} 2023-04-13 20:54:46,790 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:54:46,790 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 12f4a42416bb74102917a94867a06dfb, NAME => 'TestAcidGuarantees,,1681419251890.12f4a42416bb74102917a94867a06dfb.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:54:46,790 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:54:46,790 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419286790"}]},"ts":"9223372036854775807"} 2023-04-13 20:54:46,791 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:54:46,793 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,794 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-13 20:54:46,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-13 20:54:46,833 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-13 20:54:46,846 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 230), OpenFileDescriptor=522 (was 548), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 383), ProcessCount=170 (was 170), AvailableMemoryMB=631 (was 845) 2023-04-13 20:54:46,858 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=522, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=170, AvailableMemoryMB=631 2023-04-13 20:54:46,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:54:46,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:54:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:54:46,862 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:54:46,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-13 20:54:46,863 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:54:46,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:46,864 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:46,865 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 empty. 2023-04-13 20:54:46,865 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:46,865 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:54:46,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:47,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:47,272 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:54:47,274 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5c92cecd6957c182fb4d0c5ce319c122, NAME => 'TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:54:47,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5c92cecd6957c182fb4d0c5ce319c122, disabling compactions & flushes 2023-04-13 20:54:47,680 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. after waiting 0 ms 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:47,680 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:47,680 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:47,683 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:54:47,683 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419287683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419287683"}]},"ts":"1681419287683"} 2023-04-13 20:54:47,684 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:54:47,685 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:54:47,685 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419287685"}]},"ts":"1681419287685"} 2023-04-13 20:54:47,686 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:54:47,690 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, ASSIGN}] 2023-04-13 20:54:47,691 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, ASSIGN 2023-04-13 20:54:47,692 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:54:47,843 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5c92cecd6957c182fb4d0c5ce319c122, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:47,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419287843"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419287843"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419287843"}]},"ts":"1681419287843"} 2023-04-13 20:54:47,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:54:47,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:47,999 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:47,999 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5c92cecd6957c182fb4d0c5ce319c122, NAME => 'TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:54:48,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:54:48,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,000 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,001 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,003 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A 2023-04-13 20:54:48,003 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A 2023-04-13 20:54:48,003 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:48,003 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c92cecd6957c182fb4d0c5ce319c122 columnFamilyName A 2023-04-13 20:54:48,004 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(310): Store=5c92cecd6957c182fb4d0c5ce319c122/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:48,004 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,005 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B 2023-04-13 20:54:48,005 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B 2023-04-13 20:54:48,005 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:48,005 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c92cecd6957c182fb4d0c5ce319c122 columnFamilyName B 2023-04-13 20:54:48,006 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(310): Store=5c92cecd6957c182fb4d0c5ce319c122/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:48,006 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,007 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C 2023-04-13 20:54:48,007 DEBUG [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C 2023-04-13 20:54:48,007 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:54:48,007 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c92cecd6957c182fb4d0c5ce319c122 columnFamilyName C 2023-04-13 20:54:48,008 INFO [StoreOpener-5c92cecd6957c182fb4d0c5ce319c122-1] regionserver.HStore(310): Store=5c92cecd6957c182fb4d0c5ce319c122/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:54:48,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:48,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:54:48,011 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:48,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:54:48,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5c92cecd6957c182fb4d0c5ce319c122; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60881109, jitterRate=-0.09280078113079071}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:54:48,013 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:48,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., pid=30, masterSystemTime=1681419287995 2023-04-13 20:54:48,015 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:48,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:48,016 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5c92cecd6957c182fb4d0c5ce319c122, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:48,016 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419288015"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419288015"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419288015"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419288015"}]},"ts":"1681419288015"} 2023-04-13 20:54:48,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-13 20:54:48,018 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 in 173 msec 2023-04-13 20:54:48,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-13 20:54:48,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, ASSIGN in 328 msec 2023-04-13 20:54:48,020 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:54:48,020 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419288020"}]},"ts":"1681419288020"} 2023-04-13 20:54:48,021 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:54:48,023 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:54:48,024 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1630 sec 2023-04-13 20:54:48,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-13 20:54:48,967 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-13 20:54:48,968 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:48,973 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:48,974 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:48,975 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:48,976 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:54:48,978 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57540, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:54:48,979 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:48,983 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:48,983 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:48,987 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:48,988 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:48,991 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:48,992 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:48,996 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:48,996 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,000 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,000 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,005 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,005 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,008 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,009 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,012 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,012 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,015 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,016 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x6f8ff99f to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:54:49,020 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d2b992d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:54:49,024 DEBUG [hconnection-0x252cfa9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,024 DEBUG [hconnection-0x5c11121c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,024 DEBUG [hconnection-0x455dcdc4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,025 DEBUG [hconnection-0x739e8592-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,026 DEBUG [hconnection-0x7e4ca863-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:49,026 DEBUG [hconnection-0x10493efe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,026 DEBUG [hconnection-0x2e6387ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,027 DEBUG [hconnection-0x55993ed3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,027 DEBUG [hconnection-0xec34be8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,027 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48270, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,027 DEBUG [hconnection-0x5ce72d93-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:54:49,028 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48290, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,029 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48324, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,032 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48256, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,033 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48248, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,034 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48278, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,034 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48272, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,034 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48310, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,034 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,035 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:54:49,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:49,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:49,036 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:49,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:49,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:49,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:49,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:49,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:49,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:49,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:49,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:49,049 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,050 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:49,052 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:49,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:49,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:49,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:49,068 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/d212c95abeda46b2ba7bc5b775454a48 2023-04-13 20:54:49,082 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/43fa791836ad484da811bd790c41cb10 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419289065, max timestamp=1681419289080Num uniques 859; 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419289065, max timestamp=1681419289074Num uniques 859; 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419289065, max timestamp=1681419289080Num uniques 859; 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:49,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:49,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419289137, max timestamp=1681419289142Num uniques 1018; 2023-04-13 20:54:49,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419289137, max timestamp=1681419289142Num uniques 1018; 2023-04-13 20:54:49,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 20:54:49,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 20:54:49,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289137, max timestamp=1681419289142Num uniques 1018; 2023-04-13 20:54:49,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-04-13 20:54:49,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419289156, max timestamp=1681419289161Num uniques 1077; 2023-04-13 20:54:49,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 20:54:49,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419289156, max timestamp=1681419289161Num uniques 1077; 2023-04-13 20:54:49,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-13 20:54:49,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419289156, max timestamp=1681419289161Num uniques 1009; 2023-04-13 20:54:49,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419289173, max timestamp=1681419289177Num uniques 1036; 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1681419289169, max timestamp=1681419289177Num uniques 1036; 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419289173, max timestamp=1681419289177Num uniques 1036; 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 20:54:49,180 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-04-13 20:54:49,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419289193, max timestamp=1681419289197Num uniques 1145; 2023-04-13 20:54:49,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-04-13 20:54:49,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289193, max timestamp=1681419289197Num uniques 1045; 2023-04-13 20:54:49,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47852, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-04-13 20:54:49,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289193, max timestamp=1681419289197Num uniques 1045; 2023-04-13 20:54:49,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-04-13 20:54:49,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419289210, max timestamp=1681419289214Num uniques 1009; 2023-04-13 20:54:49,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46218, new segment data size=7050, suffix heap size=132904,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=154 2023-04-13 20:54:49,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419289210, max timestamp=1681419289214Num uniques 1054; 2023-04-13 20:54:49,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419289210, max timestamp=1681419289214Num uniques 1009; 2023-04-13 20:54:49,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 20:54:49,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=7050, suffix heap size=132904,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=154 2023-04-13 20:54:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419289225, max timestamp=1681419289228Num uniques 1013; 2023-04-13 20:54:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419289225, max timestamp=1681419289228Num uniques 1013; 2023-04-13 20:54:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:54:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:54:49,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419289225, max timestamp=1681419289228Num uniques 1013; 2023-04-13 20:54:49,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:54:49,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419289239, max timestamp=1681419289243Num uniques 1072; 2023-04-13 20:54:49,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:49,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10159, totalHeapSize=23328, min timestamp=1681419289239, max timestamp=1681419289243Num uniques 1072; 2023-04-13 20:54:49,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419289239, max timestamp=1681419289243Num uniques 1072; 2023-04-13 20:54:49,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49089, new segment data size=10159, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:49,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:54:49,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419289248, max timestamp=1681419289262Num uniques 1081; 2023-04-13 20:54:49,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 20:54:49,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419289248, max timestamp=1681419289262Num uniques 1081; 2023-04-13 20:54:49,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 20:54:49,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419289248, max timestamp=1681419289262Num uniques 1081; 2023-04-13 20:54:49,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 20:54:49,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681419289276, max timestamp=1681419289283Num uniques 1090; 2023-04-13 20:54:49,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-13 20:54:49,285 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681419289276, max timestamp=1681419289283Num uniques 1090; 2023-04-13 20:54:49,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1681419289276, max timestamp=1681419289283Num uniques 1090; 2023-04-13 20:54:49,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-13 20:54:49,286 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-04-13 20:54:49,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419349296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13684, totalHeapSize=31336, min timestamp=1681419289291, max timestamp=1681419289297Num uniques 1149; 2023-04-13 20:54:49,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13684, totalHeapSize=31336, min timestamp=1681419289291, max timestamp=1681419289297Num uniques 1149; 2023-04-13 20:54:49,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=13684, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-13 20:54:49,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13685, totalHeapSize=31336, min timestamp=1681419289291, max timestamp=1681419289297Num uniques 1149; 2023-04-13 20:54:49,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52615, new segment data size=13685, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-13 20:54:49,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52614, new segment data size=13684, suffix heap size=147464,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=299 2023-04-13 20:54:49,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419289309, max timestamp=1681419289314Num uniques 1158; 2023-04-13 20:54:49,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419349313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-13 20:54:49,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419289309, max timestamp=1681419289314Num uniques 1158; 2023-04-13 20:54:49,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-13 20:54:49,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419289309, max timestamp=1681419289314Num uniques 1158; 2023-04-13 20:54:49,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-04-13 20:54:49,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419289323, max timestamp=1681419289332Num uniques 1117; 2023-04-13 20:54:49,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:54:49,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419289323, max timestamp=1681419289332Num uniques 1117; 2023-04-13 20:54:49,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:54:49,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419289323, max timestamp=1681419289332Num uniques 1117; 2023-04-13 20:54:49,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:54:49,351 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8055, totalHeapSize=18544, min timestamp=1681419289345, max timestamp=1681419289349Num uniques 1076; 2023-04-13 20:54:49,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681419289345, max timestamp=1681419289349Num uniques 1076; 2023-04-13 20:54:49,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49275, new segment data size=8055, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:54:49,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1681419289345, max timestamp=1681419289349Num uniques 1076; 2023-04-13 20:54:49,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49266, new segment data size=8046, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:54:49,352 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49266, new segment data size=8046, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:54:49,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681419289362, max timestamp=1681419289366Num uniques 1035; 2023-04-13 20:54:49,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-13 20:54:49,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681419289362, max timestamp=1681419289366Num uniques 1035; 2023-04-13 20:54:49,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-13 20:54:49,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1681419289362, max timestamp=1681419289366Num uniques 1035; 2023-04-13 20:54:49,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-13 20:54:49,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419289376, max timestamp=1681419289383Num uniques 1094; 2023-04-13 20:54:49,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-13 20:54:49,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419289376, max timestamp=1681419289383Num uniques 1094; 2023-04-13 20:54:49,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-13 20:54:49,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419289376, max timestamp=1681419289383Num uniques 1094; 2023-04-13 20:54:49,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-13 20:54:49,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419349401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681419289394, max timestamp=1681419289400Num uniques 1103; 2023-04-13 20:54:49,403 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-13 20:54:49,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289397, max timestamp=1681419289400Num uniques 1103; 2023-04-13 20:54:49,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=6870, suffix heap size=142680,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=150 2023-04-13 20:54:49,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289397, max timestamp=1681419289400Num uniques 1103; 2023-04-13 20:54:49,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=6870, suffix heap size=142680,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=150 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419289439, max timestamp=1681419289442Num uniques 1009; 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=7412, suffix heap size=132904,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=162 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419289439, max timestamp=1681419289442Num uniques 1062; 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419289439, max timestamp=1681419289442Num uniques 1009; 2023-04-13 20:54:49,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46207, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7820, totalHeapSize=18024, min timestamp=1681419289476, max timestamp=1681419289480Num uniques 1021; 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681419289476, max timestamp=1681419289480Num uniques 1021; 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681419289476, max timestamp=1681419289480Num uniques 1009; 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7826, suffix heap size=132904,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=171 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46750, new segment data size=7820, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 20:54:49,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 20:54:49,493 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/28a0d7587e1a4000a571e76b3a2569bb 2023-04-13 20:54:49,498 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/d212c95abeda46b2ba7bc5b775454a48 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48 2023-04-13 20:54:49,503 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48, entries=100, sequenceid=7, filesize=9.3 K 2023-04-13 20:54:49,504 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/43fa791836ad484da811bd790c41cb10 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10 2023-04-13 20:54:49,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419289489, max timestamp=1681419289505Num uniques 1030; 2023-04-13 20:54:49,507 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 20:54:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419289489, max timestamp=1681419289505Num uniques 1030; 2023-04-13 20:54:49,510 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 20:54:49,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419289489, max timestamp=1681419289505Num uniques 1030; 2023-04-13 20:54:49,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 20:54:49,513 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10, entries=100, sequenceid=7, filesize=9.3 K 2023-04-13 20:54:49,514 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/28a0d7587e1a4000a571e76b3a2569bb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb 2023-04-13 20:54:49,518 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb, entries=100, sequenceid=7, filesize=9.3 K 2023-04-13 20:54:49,520 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 468ms, sequenceid=7, compaction requested=false 2023-04-13 20:54:49,520 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:49,520 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:49,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:49,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:49,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=87.22 KB heapSize=243.56 KB 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:49,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:49,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ec4b2c0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:49,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:49,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,526 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:49,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:49,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:49,529 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:49,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:49,533 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:49,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:49,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:49,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:49,533 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:49,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419349535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:49,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:49,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419349535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,541 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:49,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419349536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,541 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:49,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419349535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419349537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f 2023-04-13 20:54:49,641 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:49,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:49,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419349641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419349642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419349642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419349642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419349643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:49,645 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:49,645 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,645 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:49,645 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:49,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,647 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:49,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:49,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:49,648 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:49,649 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,653 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:49,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:49,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1ad6bb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:49,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:49,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:49,658 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:49,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:49,661 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:49,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:49,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:49,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:49,661 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:49,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:49,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,669 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,669 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:49,670 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:49,770 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:49,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:49,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:49,773 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:49,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:49,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:49,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,776 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:49,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:49,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:49,778 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,778 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:49,778 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:49,778 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:49,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:49,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6856d9ac[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:49,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:49,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:49,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:49,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:49,788 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:49,791 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:49,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:49,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:49,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:49,791 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:49,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:49,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,798 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:49,798 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419349845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419349845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419349845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419349846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:49,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419349847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,898 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:49,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:49,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:49,902 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:49,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:49,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:49,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:49,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:49,906 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,906 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:49,906 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:49,907 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,909 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:49,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:49,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391edc8b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:49,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:49,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,912 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:49,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:49,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:49,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:49,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,913 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:49,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:49,916 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:49,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:49,917 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:49,917 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:49,917 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:49,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:49,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:49,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,921 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:49,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:49,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,923 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:49,923 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:49,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:49,923 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:49,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:49,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:49,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:49,924 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:49,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/169b779119be4e29aa4ead7795fd3fc5 2023-04-13 20:54:50,024 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:50,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:50,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:50,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:50,027 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:50,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:50,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:50,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:50,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:50,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:50,030 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:50,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:50,031 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:50,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:50,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72872bda[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:50,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:50,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:50,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:50,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:50,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:50,044 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:50,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:50,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:50,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:50,044 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:50,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,047 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:50,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:50,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,051 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:50,051 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:50,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419350148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419350148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419350149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,151 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:50,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419350149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:50,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419350150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:50,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:50,155 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:50,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:50,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:50,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:50,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:50,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:50,158 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:50,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:50,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:50,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e84843f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:50,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:50,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:50,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:50,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:50,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:50,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:50,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:50,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:50,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:50,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:50,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:50,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:50,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,174 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,174 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,274 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:50,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:50,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:50,278 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:50,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:50,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:50,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,279 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:50,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:50,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:50,281 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,281 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:50,281 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:50,281 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:50,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:50,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,285 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:50,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c1bdfa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:50,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:50,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:50,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:50,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:50,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:50,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:50,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:50,292 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:50,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:50,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:50,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:50,292 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:50,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:50,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,300 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:50,300 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=365 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59 2023-04-13 20:54:50,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f 2023-04-13 20:54:50,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f, entries=150, sequenceid=365, filesize=11.9 K 2023-04-13 20:54:50,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/169b779119be4e29aa4ead7795fd3fc5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5 2023-04-13 20:54:50,400 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:50,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:50,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:50,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5, entries=150, sequenceid=365, filesize=11.9 K 2023-04-13 20:54:50,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59 2023-04-13 20:54:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:50,408 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:50,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:50,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:50,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59, entries=150, sequenceid=365, filesize=11.9 K 2023-04-13 20:54:50,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:50,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~283.36 KB/290160, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 891ms, sequenceid=365, compaction requested=false 2023-04-13 20:54:50,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:50,411 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:50,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:50,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:50,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:50,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:50,412 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:54:50,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:50,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:50,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:50,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,424 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c62b596dc07c47319325399fff903228 2023-04-13 20:54:50,434 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/056cfa3ca15a4bd780605f72804344f4 2023-04-13 20:54:50,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:50,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419350660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419350660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419350662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419350662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419350662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419350765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419350765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419350766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419350767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419350767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,848 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/9cf7ba9e386442a4ad3e42672f5d7e23 2023-04-13 20:54:50,853 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c62b596dc07c47319325399fff903228 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228 2023-04-13 20:54:50,857 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228, entries=150, sequenceid=380, filesize=11.9 K 2023-04-13 20:54:50,858 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/056cfa3ca15a4bd780605f72804344f4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4 2023-04-13 20:54:50,865 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4, entries=150, sequenceid=380, filesize=11.9 K 2023-04-13 20:54:50,866 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/9cf7ba9e386442a4ad3e42672f5d7e23 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23 2023-04-13 20:54:50,869 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23, entries=150, sequenceid=380, filesize=11.9 K 2023-04-13 20:54:50,870 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5c92cecd6957c182fb4d0c5ce319c122 in 458ms, sequenceid=380, compaction requested=true 2023-04-13 20:54:50,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:50,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:50,871 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:50,872 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:50,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f74c3ed[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:50,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:50,874 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,875 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:50,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:50,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:50,877 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:50,877 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:50,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:50,880 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:50,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:50,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:50,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:50,880 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:50,882 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,882 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,882 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:50,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:50,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:50,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:50,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:50,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,890 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:50,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:50,890 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:50,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:50,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:50,969 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:50,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419350976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419350976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419350977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419350980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419350981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,990 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:50,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:50,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:50,993 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,996 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:50,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:50,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:50,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:50,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:50,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:50,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:50,997 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:50,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:50,999 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,999 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:50,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e6e822[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,003 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,014 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,014 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419351086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419351086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419351086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,089 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419351086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419351086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,114 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,117 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,121 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb931fd[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,126 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,130 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,130 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,130 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,130 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,130 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,131 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,131 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,131 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,132 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,135 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,136 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,136 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,236 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,239 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,239 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,239 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,239 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,242 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,242 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@739a6dbd[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,249 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,249 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,249 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,250 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,253 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,253 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,256 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,263 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:51,263 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,263 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,263 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,263 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419351289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419351290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419351290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419351290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419351290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,363 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,366 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,367 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,367 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,368 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,370 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,370 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,370 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,370 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,371 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,371 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a18b388[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,376 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/667c3742ab9d41cc8c85cb4095a7319e 2023-04-13 20:54:51,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,380 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,380 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,382 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,382 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,382 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,382 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,388 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,388 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,388 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,489 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,492 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,494 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,495 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,495 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,496 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,496 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,497 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,497 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,498 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@298cef11[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,500 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,500 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,501 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,503 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,505 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,505 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,509 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,510 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,512 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,512 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,513 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,513 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:51,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419351591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419351593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419351593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419351593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:51,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419351595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,613 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,616 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,617 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,617 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,618 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,618 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,618 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,618 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,620 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,620 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,621 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e8d0504[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,623 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,625 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,626 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,629 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,629 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,632 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,636 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,637 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,637 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,688 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:54:51,737 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,740 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,750 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,752 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,753 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,753 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e5a74ed[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,754 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,756 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,759 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,759 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,763 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,765 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,766 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,766 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,766 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/6e6b7677df464777b64ac173911db907 2023-04-13 20:54:51,866 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,869 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,869 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,869 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,869 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,871 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,871 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,871 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,871 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,873 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,873 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,876 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:51,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6101045d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:51,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,878 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:51,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:51,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,880 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:51,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:51,882 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:51,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:51,882 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:51,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,883 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,888 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,888 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,888 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,888 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:51,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:51,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:51,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,889 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:51,889 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:51,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:51,989 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:51,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:51,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:51,993 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:51,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:51,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:51,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,994 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:51,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:51,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:51,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:51,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:51,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:51,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:51,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:51,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:51,997 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:51,998 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:51,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:51,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbb8b0b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,001 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,003 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,007 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,007 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,007 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,012 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,012 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,012 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,012 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,012 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:52,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419352093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:52,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419352096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:52,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419352097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:52,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419352097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:52,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419352101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,112 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,115 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,117 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,119 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,119 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,119 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:52,119 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,119 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,124 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59681edd[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,126 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,131 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,131 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,131 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,133 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,133 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,133 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,135 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,138 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,138 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,138 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,138 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,138 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/2f42104c0ec04d05b483b4bbef864d93 2023-04-13 20:54:52,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/667c3742ab9d41cc8c85cb4095a7319e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e 2023-04-13 20:54:52,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e, entries=200, sequenceid=398, filesize=14.3 K 2023-04-13 20:54:52,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/6e6b7677df464777b64ac173911db907 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907 2023-04-13 20:54:52,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907, entries=150, sequenceid=398, filesize=11.9 K 2023-04-13 20:54:52,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/2f42104c0ec04d05b483b4bbef864d93 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93 2023-04-13 20:54:52,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93, entries=150, sequenceid=398, filesize=11.9 K 2023-04-13 20:54:52,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 1250ms, sequenceid=398, compaction requested=true 2023-04-13 20:54:52,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:52,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:52,219 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:52,219 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:52,219 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:52,221 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:52,221 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46100 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:52,221 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:54:52,221 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:54:52,221 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,221 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,221 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=47.4 K 2023-04-13 20:54:52,221 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=45.0 K 2023-04-13 20:54:52,221 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d212c95abeda46b2ba7bc5b775454a48, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681419289042 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 43fa791836ad484da811bd790c41cb10, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681419289042 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ffbbaf2e2dab46b2a5cf4bc5f094c79f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419289510 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 169b779119be4e29aa4ead7795fd3fc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419289510 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c62b596dc07c47319325399fff903228, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681419289532 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 056cfa3ca15a4bd780605f72804344f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681419289532 2023-04-13 20:54:52,222 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 667c3742ab9d41cc8c85cb4095a7319e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681419290659 2023-04-13 20:54:52,223 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6e6b7677df464777b64ac173911db907, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681419290659 2023-04-13 20:54:52,230 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:52,232 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#234 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:52,238 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,241 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,242 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,245 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:52,245 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:52,246 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:52,252 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/466a6db429314631b41331253a0bde88 2023-04-13 20:54:52,269 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/1d8149d3644146d0b4f38c5102f84b40 2023-04-13 20:54:52,286 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e112e7c712594dd88302726204c6de67 2023-04-13 20:54:52,290 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/466a6db429314631b41331253a0bde88 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88 2023-04-13 20:54:52,294 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88, entries=150, sequenceid=413, filesize=11.9 K 2023-04-13 20:54:52,295 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/1d8149d3644146d0b4f38c5102f84b40 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40 2023-04-13 20:54:52,299 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40, entries=150, sequenceid=413, filesize=11.9 K 2023-04-13 20:54:52,300 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e112e7c712594dd88302726204c6de67 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67 2023-04-13 20:54:52,306 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67, entries=150, sequenceid=413, filesize=11.9 K 2023-04-13 20:54:52,307 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 5c92cecd6957c182fb4d0c5ce319c122 in 62ms, sequenceid=413, compaction requested=true 2023-04-13 20:54:52,307 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,307 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,309 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3478e401[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,313 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,313 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,313 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,318 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,318 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,325 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,325 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,425 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,429 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,433 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,433 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,433 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,433 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,435 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db22d3f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,438 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,439 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,442 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,442 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,449 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,449 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,549 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,555 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,555 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,555 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e220651[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,565 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,571 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,571 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,577 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,577 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,643 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/6e2e97bb34a845e58ca69a5f34e6d5de as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/6e2e97bb34a845e58ca69a5f34e6d5de 2023-04-13 20:54:52,645 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/98878c0179464da5b06e69e4e7852e5b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/98878c0179464da5b06e69e4e7852e5b 2023-04-13 20:54:52,650 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into 98878c0179464da5b06e69e4e7852e5b(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:52,650 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,650 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=12, startTime=1681419292219; duration=0sec 2023-04-13 20:54:52,650 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:52,650 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58286 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:54:52,653 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,653 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=56.9 K 2023-04-13 20:54:52,653 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into 6e2e97bb34a845e58ca69a5f34e6d5de(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:52,653 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,653 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=12, startTime=1681419292219; duration=0sec 2023-04-13 20:54:52,654 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:52,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 28a0d7587e1a4000a571e76b3a2569bb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681419289042 2023-04-13 20:54:52,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1d7e3a2e2dcf46d8b1e1c0a68eeefb59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=365, earliestPutTs=1681419289510 2023-04-13 20:54:52,654 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9cf7ba9e386442a4ad3e42672f5d7e23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1681419289532 2023-04-13 20:54:52,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2f42104c0ec04d05b483b4bbef864d93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681419290659 2023-04-13 20:54:52,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e112e7c712594dd88302726204c6de67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681419290976 2023-04-13 20:54:52,664 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#238 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:52,674 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/90aca0560a7d4285af1adcb87b3a5f33 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/90aca0560a7d4285af1adcb87b3a5f33 2023-04-13 20:54:52,677 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,679 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 90aca0560a7d4285af1adcb87b3a5f33(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:52,679 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,679 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=11, startTime=1681419292219; duration=0sec 2023-04-13 20:54:52,679 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:52,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,681 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,683 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,684 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,684 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,685 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,686 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,686 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,688 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,688 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37257bbe[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,691 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,695 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,695 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,703 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,704 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,704 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,704 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,804 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,808 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,810 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448ec9b7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,819 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,820 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,823 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,823 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,823 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,823 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,823 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,824 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,824 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,830 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,830 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:52,930 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:52,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:52,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:52,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:52,934 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:52,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,936 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,936 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,937 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,937 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:52,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:52,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:52,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:52,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:52,939 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:52,940 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@819dfcc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:52,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:52,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:52,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:52,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:52,956 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:52,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:52,959 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:52,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:52,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:52,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:52,959 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:52,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:52,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:52,964 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,965 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:52,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:52,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:52,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:52,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:52,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:52,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:52,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:52,968 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:52,969 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,069 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,072 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,074 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,076 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,076 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,076 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:53,076 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36b75010[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,084 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,084 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,087 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,087 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:53,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,096 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,096 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:53,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:53,112 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,118 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/54855fba22454e96905665744260956a 2023-04-13 20:54:53,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419353124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419353124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419353125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419353125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419353126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,196 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,200 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,204 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,204 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,204 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,204 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5325f3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,214 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,214 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,215 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,215 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,215 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,220 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:53,220 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,220 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419353229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419353229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419353230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419353230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419353230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,320 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,323 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,323 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,323 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,326 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,326 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,326 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,327 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,327 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,327 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,327 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,327 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822927f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,331 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,331 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,333 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,333 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,333 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,334 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,341 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,341 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,343 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,345 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,351 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,352 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,352 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48328 deadline: 1681419353432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419353433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419353433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419353433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419353433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,452 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,455 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,456 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,456 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,458 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,458 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,459 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,459 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,459 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,459 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,461 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,461 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e83d908[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,464 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,465 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,469 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,470 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,472 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,477 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,477 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/7d9b85a68f8e4c5fa2233739c396166e 2023-04-13 20:54:53,577 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,580 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,585 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,585 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,585 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,587 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,587 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52382518[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,593 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,595 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,596 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,600 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,603 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,603 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,703 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,706 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,711 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,711 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,711 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,711 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61f7ee11[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,717 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,720 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,720 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,726 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,726 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419353735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48272 deadline: 1681419353736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419353736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419353737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:53,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48324 deadline: 1681419353737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,826 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,829 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:53,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:53,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:53,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:53,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:53,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:53,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dca293a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:53,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,841 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:53,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:53,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:53,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:53,846 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:53,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:53,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:53,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:53,846 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:53,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:53,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:53,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:53,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:53,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,853 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:53,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:53,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:53,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:53,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:53,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,854 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:53,854 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:53,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/c54708ca4a354d6d9653aef5af4eb855 2023-04-13 20:54:53,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/54855fba22454e96905665744260956a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a 2023-04-13 20:54:53,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a, entries=200, sequenceid=435, filesize=14.3 K 2023-04-13 20:54:53,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/7d9b85a68f8e4c5fa2233739c396166e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e 2023-04-13 20:54:53,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e, entries=150, sequenceid=435, filesize=11.9 K 2023-04-13 20:54:53,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/c54708ca4a354d6d9653aef5af4eb855 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855 2023-04-13 20:54:53,954 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:53,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:53,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855, entries=150, sequenceid=435, filesize=11.9 K 2023-04-13 20:54:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:53,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:53,958 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:53,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:53,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:53,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5c92cecd6957c182fb4d0c5ce319c122 in 848ms, sequenceid=435, compaction requested=true 2023-04-13 20:54:53,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:53,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,958 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:53,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:53,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:53,958 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:53,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:53,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:53,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:54:53,960 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,960 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:53,960 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/6e2e97bb34a845e58ca69a5f34e6d5de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=38.2 K 2023-04-13 20:54:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:53,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36696 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:54:53,961 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,961 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/98878c0179464da5b06e69e4e7852e5b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=35.8 K 2023-04-13 20:54:53,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6e2e97bb34a845e58ca69a5f34e6d5de, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681419290659 2023-04-13 20:54:53,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:53,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 98878c0179464da5b06e69e4e7852e5b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1681419290659 2023-04-13 20:54:53,961 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:53,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:53,962 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 466a6db429314631b41331253a0bde88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681419290976 2023-04-13 20:54:53,962 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1d8149d3644146d0b4f38c5102f84b40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681419290976 2023-04-13 20:54:53,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:53,962 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 54855fba22454e96905665744260956a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681419293109 2023-04-13 20:54:53,962 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7d9b85a68f8e4c5fa2233739c396166e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681419293109 2023-04-13 20:54:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:53,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:53,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:53,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:53,964 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:53,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,965 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:53,965 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:53,970 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#243 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:53,971 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#244 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:53,980 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/1f8a6b7849b04fd59a4ea056dc8d3d6b 2023-04-13 20:54:53,990 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/b5c3582d1fed40109efdffd77577a1fd 2023-04-13 20:54:53,999 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e433ce481b0d4d2c841fc3e7551a3053 2023-04-13 20:54:54,003 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/1f8a6b7849b04fd59a4ea056dc8d3d6b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b 2023-04-13 20:54:54,007 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b, entries=150, sequenceid=454, filesize=11.9 K 2023-04-13 20:54:54,008 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/b5c3582d1fed40109efdffd77577a1fd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd 2023-04-13 20:54:54,011 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd, entries=150, sequenceid=454, filesize=11.9 K 2023-04-13 20:54:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e433ce481b0d4d2c841fc3e7551a3053 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053 2023-04-13 20:54:54,017 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053, entries=150, sequenceid=454, filesize=11.9 K 2023-04-13 20:54:54,018 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 5c92cecd6957c182fb4d0c5ce319c122 in 54ms, sequenceid=454, compaction requested=true 2023-04-13 20:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,018 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d41f612[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,024 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,029 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,029 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,037 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:54,037 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,137 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,140 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,145 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ec21647[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,148 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,154 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,154 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,160 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,161 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:54,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:54,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:54,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:54,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:54,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:54,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:54,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:54,261 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419294254, max timestamp=1681419294261Num uniques 859; 2023-04-13 20:54:54,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:54:54,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419294254, max timestamp=1681419294261Num uniques 859; 2023-04-13 20:54:54,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:54:54,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1681419294254, max timestamp=1681419294261Num uniques 859; 2023-04-13 20:54:54,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-04-13 20:54:54,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,266 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,267 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,269 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,269 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,269 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,269 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:54,269 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,276 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@601bf0e8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419354276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419354276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,280 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419354278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419354278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419354279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,284 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,284 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,291 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,291 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,291 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419354380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419354381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,385 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cfd901160bf54577a3e7d4cbcf701dbf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cfd901160bf54577a3e7d4cbcf701dbf 2023-04-13 20:54:54,385 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/d0f33fab4745491ead6b10b3522db6ef as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d0f33fab4745491ead6b10b3522db6ef 2023-04-13 20:54:54,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419354384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419354384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419354384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,389 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into cfd901160bf54577a3e7d4cbcf701dbf(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:54,390 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,390 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=13, startTime=1681419293958; duration=0sec 2023-04-13 20:54:54,390 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:54,390 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:54,390 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into d0f33fab4745491ead6b10b3522db6ef(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:54,391 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,391 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=13, startTime=1681419293958; duration=0sec 2023-04-13 20:54:54,391 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:54,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:54,391 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:54:54,391 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,391 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/90aca0560a7d4285af1adcb87b3a5f33, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=35.9 K 2023-04-13 20:54:54,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,392 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 90aca0560a7d4285af1adcb87b3a5f33, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1681419290976 2023-04-13 20:54:54,392 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c54708ca4a354d6d9653aef5af4eb855, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681419293109 2023-04-13 20:54:54,392 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e433ce481b0d4d2c841fc3e7551a3053, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681419293123 2023-04-13 20:54:54,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,394 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,399 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#248 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:54,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ca48628[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,415 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,415 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,421 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,421 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,521 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,525 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,526 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,527 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,527 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:54,529 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,531 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef71ac8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,536 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,539 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,539 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,547 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,547 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,547 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419354584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419354585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419354587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419354588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419354588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,647 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,650 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/1db8ee8bff9c4bec81e0031b95a98d56 2023-04-13 20:54:54,652 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,652 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,653 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,653 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:54,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:54,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:54,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a95b7f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:54,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,660 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:54,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:54,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:54,662 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/abab2c24ab454ee0a05781be24fbf1f1 2023-04-13 20:54:54,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:54,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:54,668 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:54,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:54,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:54,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:54,668 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:54,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:54,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:54,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/99fadd37c0914d03b63c5af79536aa76 2023-04-13 20:54:54,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:54,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:54,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:54,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:54,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:54,676 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:54,676 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:54,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:54,676 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:54,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/1db8ee8bff9c4bec81e0031b95a98d56 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56 2023-04-13 20:54:54,680 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56, entries=200, sequenceid=465, filesize=14.3 K 2023-04-13 20:54:54,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/abab2c24ab454ee0a05781be24fbf1f1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1 2023-04-13 20:54:54,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1, entries=150, sequenceid=465, filesize=11.9 K 2023-04-13 20:54:54,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/99fadd37c0914d03b63c5af79536aa76 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76 2023-04-13 20:54:54,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76, entries=150, sequenceid=465, filesize=11.9 K 2023-04-13 20:54:54,690 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5c92cecd6957c182fb4d0c5ce319c122 in 446ms, sequenceid=465, compaction requested=true 2023-04-13 20:54:54,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:54,690 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:54,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:54,690 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:54:54,691 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:54,691 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:54:54,691 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,691 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d0f33fab4745491ead6b10b3522db6ef, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=38.3 K 2023-04-13 20:54:54,692 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d0f33fab4745491ead6b10b3522db6ef, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681419293109 2023-04-13 20:54:54,692 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1f8a6b7849b04fd59a4ea056dc8d3d6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681419293123 2023-04-13 20:54:54,693 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1db8ee8bff9c4bec81e0031b95a98d56, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681419294242 2023-04-13 20:54:54,699 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#251 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:54,776 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:54,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:54,780 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:54,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:54,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:54,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,782 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:54,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:54,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:54,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:54,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:54,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:54,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:54,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:54,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:54,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:54,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:54,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:54,784 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:54,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:54,785 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:54,792 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/63c480b575eb41d48f5de665a2ee78f9 2023-04-13 20:54:54,803 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cb12e0debca64314b2d0e67c4885b7c4 2023-04-13 20:54:54,818 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e81895fb25164ebe897e389e41244c94 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e81895fb25164ebe897e389e41244c94 2023-04-13 20:54:54,822 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into e81895fb25164ebe897e389e41244c94(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:54,822 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,822 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=13, startTime=1681419293959; duration=0sec 2023-04-13 20:54:54,822 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:54:54,822 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:54,823 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:54,823 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:54:54,823 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:54,823 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cfd901160bf54577a3e7d4cbcf701dbf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=35.9 K 2023-04-13 20:54:54,823 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cfd901160bf54577a3e7d4cbcf701dbf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1681419293109 2023-04-13 20:54:54,824 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b5c3582d1fed40109efdffd77577a1fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681419293123 2023-04-13 20:54:54,824 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting abab2c24ab454ee0a05781be24fbf1f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681419294242 2023-04-13 20:54:54,833 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:54,842 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/af4f1ad4d37149c18683c17ae1507359 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/af4f1ad4d37149c18683c17ae1507359 2023-04-13 20:54:54,846 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into af4f1ad4d37149c18683c17ae1507359(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:54,846 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=13, startTime=1681419294690; duration=0sec 2023-04-13 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:54,846 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:54:54,847 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. because compaction request was cancelled 2023-04-13 20:54:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:54,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:54,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48278 deadline: 1681419354894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419354895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419354895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419354896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:54,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:54,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419354896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419354999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419355000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419355000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419355000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,108 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/aeafe16ebd2549269acf908fc74bb0f7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/aeafe16ebd2549269acf908fc74bb0f7 2023-04-13 20:54:55,112 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into aeafe16ebd2549269acf908fc74bb0f7(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:55,113 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:55,113 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=13, startTime=1681419294690; duration=0sec 2023-04-13 20:54:55,113 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:55,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419355203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419355203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,205 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419355204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419355204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,212 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/1814485227084bdeb98e8b52423b61ab 2023-04-13 20:54:55,217 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/63c480b575eb41d48f5de665a2ee78f9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9 2023-04-13 20:54:55,221 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9, entries=150, sequenceid=501, filesize=11.9 K 2023-04-13 20:54:55,222 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cb12e0debca64314b2d0e67c4885b7c4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4 2023-04-13 20:54:55,226 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4, entries=150, sequenceid=501, filesize=11.9 K 2023-04-13 20:54:55,227 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/1814485227084bdeb98e8b52423b61ab as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab 2023-04-13 20:54:55,232 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab, entries=150, sequenceid=501, filesize=11.9 K 2023-04-13 20:54:55,233 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=46.96 KB/48090 for 5c92cecd6957c182fb4d0c5ce319c122 in 449ms, sequenceid=501, compaction requested=true 2023-04-13 20:54:55,233 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:55,233 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,236 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,237 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8a6d32[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,238 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,238 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,240 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,240 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,241 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,241 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:55,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,244 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,244 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,244 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,244 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,244 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,252 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,252 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,252 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,352 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:55,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:55,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:55,356 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:55,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:55,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:55,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:55,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:55,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:55,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:55,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:55,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:55,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:55,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:55,360 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:55,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,368 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c8f5017ed4a44a6fab971c679e2bf77d 2023-04-13 20:54:55,378 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/873b2fc817ff4c9abd88947941edaf0a 2023-04-13 20:54:55,387 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=514 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/9c1a7802789544a594e95a7e32278c6d 2023-04-13 20:54:55,391 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c8f5017ed4a44a6fab971c679e2bf77d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d 2023-04-13 20:54:55,395 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d, entries=150, sequenceid=514, filesize=11.9 K 2023-04-13 20:54:55,396 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/873b2fc817ff4c9abd88947941edaf0a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a 2023-04-13 20:54:55,400 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a, entries=150, sequenceid=514, filesize=11.9 K 2023-04-13 20:54:55,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/9c1a7802789544a594e95a7e32278c6d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d 2023-04-13 20:54:55,403 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d, entries=150, sequenceid=514, filesize=11.9 K 2023-04-13 20:54:55,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:55,404 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:55,404 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for 5c92cecd6957c182fb4d0c5ce319c122 in 44ms, sequenceid=514, compaction requested=true 2023-04-13 20:54:55,404 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:55,404 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,404 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,405 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1767332d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,413 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,413 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,414 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,414 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,414 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:55,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,418 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,419 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,419 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:55,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:55,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:55,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419295462, max timestamp=1681419295475Num uniques 859; 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419295462, max timestamp=1681419295475Num uniques 859; 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419295462, max timestamp=1681419295475Num uniques 859; 2023-04-13 20:54:55,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419355509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419355510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419355512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419355513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419355513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,519 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:55,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:55,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:55,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:55,522 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:55,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:55,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:55,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,524 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:55,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:55,525 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:55,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:55,525 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,526 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:55,526 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:55,526 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,530 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51004a53[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:55,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,538 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,538 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,547 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,547 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,547 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419355614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419355614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419355615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419355617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419355617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,647 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:55,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:55,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:55,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:55,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:55,650 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:55,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:55,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:55,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,652 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,652 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,652 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,652 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:55,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:55,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:55,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:55,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:55,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:55,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4be891af[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,659 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,661 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:55,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,665 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,665 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,671 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,672 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,772 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:55,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:55,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:55,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:55,775 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:55,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:55,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:55,776 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,776 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,782 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:55,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:55,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:55,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:55,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:55,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:55,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7737961d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,790 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:55,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,793 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,793 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:55,801 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,801 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419355817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419355817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419355817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419355820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:55,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419355821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c9a220c40e654730b09f663a9e928976 2023-04-13 20:54:55,901 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:55,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:55,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:55,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:55,904 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:55,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:55,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,906 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:55,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:55,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:55,908 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:55,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:55,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:55,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e72d8d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:55,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:55,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:55,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:55,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:55,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:55,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:55,918 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:55,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:55,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:55,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:55,919 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:55,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:55,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:55,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:55,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:55,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:55,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:55,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:55,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:55,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:55,929 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,929 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:55,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:55,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:56,029 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,043 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,046 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,046 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,047 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:56,047 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:56,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:56,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae6b630[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:56,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:56,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:56,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:56,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:56,058 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,058 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:56,061 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:56,061 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:56,061 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:56,061 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:56,061 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,063 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,064 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:56,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,067 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,067 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:56,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419356120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419356121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48256 deadline: 1681419356121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419356123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419356124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,167 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,170 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:56,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:56,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:56,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:56,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c2b2a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:56,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,180 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:56,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:56,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:56,184 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:56,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:56,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:56,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:56,184 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:56,185 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,185 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,185 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,191 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:56,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,191 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:56,191 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/0564b39e02664830be6a4869d18ea7b3 2023-04-13 20:54:56,291 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,295 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:56,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:56,298 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:56,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:56,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:56,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:56,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ec0d2e8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:56,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:56,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:56,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:56,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:56,309 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:56,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:56,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:56,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:56,309 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:56,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:56,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,316 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:56,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,316 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,416 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,419 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:56,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:56,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:56,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:56,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:56,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eab15f8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:56,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:56,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,432 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:56,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:56,434 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:56,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:56,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:56,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:56,434 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:56,436 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,436 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,436 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:56,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:56,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,441 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:56,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,441 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:56,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,541 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,544 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:56,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,547 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,547 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,547 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:56,548 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:56,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:56,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:56,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,553 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:56,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:56,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1143612b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:56,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,555 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:56,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:56,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:56,557 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:56,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:56,559 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:56,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:56,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:56,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:56,559 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:56,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:56,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:56,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:56,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:56,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:56,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:56,565 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:56,565 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:56,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419356626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419356626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419356628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419356629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:56,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419356629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=524 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/a9dffaea55fd4c67bdad786a6df1a1d1 2023-04-13 20:54:56,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c9a220c40e654730b09f663a9e928976 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976 2023-04-13 20:54:56,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976, entries=150, sequenceid=524, filesize=11.9 K 2023-04-13 20:54:56,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/0564b39e02664830be6a4869d18ea7b3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3 2023-04-13 20:54:56,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3, entries=150, sequenceid=524, filesize=11.9 K 2023-04-13 20:54:56,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/a9dffaea55fd4c67bdad786a6df1a1d1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1 2023-04-13 20:54:56,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1, entries=150, sequenceid=524, filesize=11.9 K 2023-04-13 20:54:56,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5c92cecd6957c182fb4d0c5ce319c122 in 1244ms, sequenceid=524, compaction requested=true 2023-04-13 20:54:56,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:56,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:56,662 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:56,662 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:54:56,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:56,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:56,664 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:56,664 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:54:56,664 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,664 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/aeafe16ebd2549269acf908fc74bb0f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=47.9 K 2023-04-13 20:54:56,664 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aeafe16ebd2549269acf908fc74bb0f7, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681419294242 2023-04-13 20:54:56,665 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 63c480b575eb41d48f5de665a2ee78f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681419294275 2023-04-13 20:54:56,665 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c8f5017ed4a44a6fab971c679e2bf77d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681419294888 2023-04-13 20:54:56,665 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c9a220c40e654730b09f663a9e928976, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:56,666 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:56,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:56,667 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:54:56,667 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:54:56,667 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,667 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/af4f1ad4d37149c18683c17ae1507359, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=47.9 K 2023-04-13 20:54:56,668 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting af4f1ad4d37149c18683c17ae1507359, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681419294242 2023-04-13 20:54:56,668 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cb12e0debca64314b2d0e67c4885b7c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681419294275 2023-04-13 20:54:56,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:56,668 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 873b2fc817ff4c9abd88947941edaf0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681419294888 2023-04-13 20:54:56,669 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0564b39e02664830be6a4869d18ea7b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:56,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:56,670 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:56,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:56,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:56,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,672 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:56,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:56,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:56,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:56,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:56,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:56,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:56,678 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#262 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:56,686 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#263 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:56,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:56,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:56,698 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,698 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:56,698 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 20:54:56,698 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:56,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:56,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:56,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:56,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:56,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:56,748 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/f9c24648a7244c8cad06f912593854f7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f9c24648a7244c8cad06f912593854f7 2023-04-13 20:54:56,753 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into f9c24648a7244c8cad06f912593854f7(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:56,753 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:56,753 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=12, startTime=1681419296662; duration=0sec 2023-04-13 20:54:56,753 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:56,753 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:54:56,755 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61204 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:54:56,755 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:54:56,755 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:56,755 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e81895fb25164ebe897e389e41244c94, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=59.8 K 2023-04-13 20:54:56,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e81895fb25164ebe897e389e41244c94, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1681419293123 2023-04-13 20:54:56,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 99fadd37c0914d03b63c5af79536aa76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1681419294242 2023-04-13 20:54:56,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1814485227084bdeb98e8b52423b61ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1681419294275 2023-04-13 20:54:56,757 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9c1a7802789544a594e95a7e32278c6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=514, earliestPutTs=1681419294888 2023-04-13 20:54:56,757 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a9dffaea55fd4c67bdad786a6df1a1d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:56,768 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#265 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:56,782 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/60abbd41caf0477baff17d40276d0f2d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/60abbd41caf0477baff17d40276d0f2d 2023-04-13 20:54:56,787 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 60abbd41caf0477baff17d40276d0f2d(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:56,787 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:56,787 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=11, startTime=1681419296662; duration=0sec 2023-04-13 20:54:56,787 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:57,126 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/f905f7ceb2034a5c9898bc66d35573d8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f905f7ceb2034a5c9898bc66d35573d8 2023-04-13 20:54:57,130 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into f905f7ceb2034a5c9898bc66d35573d8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:57,130 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:57,130 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=12, startTime=1681419296662; duration=0sec 2023-04-13 20:54:57,131 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:57,145 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/db1c3ba832a9428cbc00e7fbe359d6ff 2023-04-13 20:54:57,154 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cf5c26c9706b43c1a4cfa16e23c99426 2023-04-13 20:54:57,565 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=561 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/338ad10df3c64568b5922bc60284f1fb 2023-04-13 20:54:57,572 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/db1c3ba832a9428cbc00e7fbe359d6ff as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff 2023-04-13 20:54:57,576 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff, entries=150, sequenceid=561, filesize=11.9 K 2023-04-13 20:54:57,577 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cf5c26c9706b43c1a4cfa16e23c99426 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426 2023-04-13 20:54:57,580 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426, entries=150, sequenceid=561, filesize=11.9 K 2023-04-13 20:54:57,581 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/338ad10df3c64568b5922bc60284f1fb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb 2023-04-13 20:54:57,585 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb, entries=150, sequenceid=561, filesize=11.9 K 2023-04-13 20:54:57,585 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 5c92cecd6957c182fb4d0c5ce319c122 in 887ms, sequenceid=561, compaction requested=false 2023-04-13 20:54:57,586 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:57,586 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:57,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:57,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:57,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75724d52[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:57,590 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:57,590 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:57,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:57,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:57,594 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:57,594 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,594 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:57,594 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:57,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:57,597 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:57,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:57,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:57,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:57,597 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:57,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:57,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:57,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,606 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,606 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,606 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:57,606 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:57,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:57,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:57,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/fb37ecac3ad54f7fbaf0e9d6fadf0452 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419297645, max timestamp=1681419297652Num uniques 859; 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419297645, max timestamp=1681419297652Num uniques 859; 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419297645, max timestamp=1681419297652Num uniques 859; 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:57,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:54:57,661 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cba518928c81432ba66363c08900f5f2 2023-04-13 20:54:57,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419357668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419357670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419357670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419357671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419357671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,706 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:57,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:57,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:57,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:57,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:57,709 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:57,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:57,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:57,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:57,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:57,717 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d6d8686[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:57,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:57,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:57,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:57,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,722 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:57,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:57,724 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:57,724 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:57,724 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:57,724 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:57,724 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:57,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,726 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:57,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,731 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,731 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:57,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419357771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419357774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419357774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419357774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419357774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,831 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:57,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:57,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:57,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:57,835 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,837 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:57,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:57,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:57,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:57,839 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:57,839 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:57,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:57,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9f5006[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:57,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:57,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,843 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,843 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:57,843 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:57,844 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:57,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,846 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:57,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:57,849 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:57,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:57,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:57,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:57,849 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:57,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,854 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:57,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,854 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:57,854 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:57,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,954 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:57,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:57,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:57,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:57,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:57,958 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:57,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:57,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:57,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:57,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:57,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:57,961 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,961 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:57,961 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:57,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df3a67[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:57,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:57,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,967 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:57,967 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:57,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:57,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:57,971 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:57,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:57,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:57,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:57,971 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:57,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:57,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,974 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:57,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419357974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,977 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,977 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:57,977 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:57,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:57,977 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:57,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:57,978 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:57,977 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:57,978 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:57,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:57,977 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:57,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:57,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:57,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:57,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:57,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419357978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419357979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419357979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:57,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:57,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419357979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=575 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/3ec36c3185a84cfbaac4b762e342aec5 2023-04-13 20:54:58,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/fb37ecac3ad54f7fbaf0e9d6fadf0452 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452 2023-04-13 20:54:58,077 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452, entries=200, sequenceid=575, filesize=14.3 K 2023-04-13 20:54:58,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cba518928c81432ba66363c08900f5f2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2 2023-04-13 20:54:58,078 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:58,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:58,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:58,081 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:58,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:58,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:58,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:58,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,082 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2, entries=150, sequenceid=575, filesize=11.9 K 2023-04-13 20:54:58,083 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/3ec36c3185a84cfbaac4b762e342aec5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5 2023-04-13 20:54:58,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:58,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:58,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:58,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:58,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:58,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:58,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5, entries=150, sequenceid=575, filesize=11.9 K 2023-04-13 20:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:58,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 5c92cecd6957c182fb4d0c5ce319c122 in 452ms, sequenceid=575, compaction requested=true 2023-04-13 20:54:58,088 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:58,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:58,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:58,088 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:58,088 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:58,088 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:58,089 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:58,089 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39476 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:58,089 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:54:58,089 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:54:58,089 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,089 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,089 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f9c24648a7244c8cad06f912593854f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.2 K 2023-04-13 20:54:58,089 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f905f7ceb2034a5c9898bc66d35573d8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=38.6 K 2023-04-13 20:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2620e7d3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:58,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:58,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,089 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f9c24648a7244c8cad06f912593854f7, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:58,090 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f905f7ceb2034a5c9898bc66d35573d8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:58,090 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cf5c26c9706b43c1a4cfa16e23c99426, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681419295509 2023-04-13 20:54:58,090 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,090 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting db1c3ba832a9428cbc00e7fbe359d6ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681419295509 2023-04-13 20:54:58,091 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cba518928c81432ba66363c08900f5f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:54:58,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:58,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:58,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:58,091 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fb37ecac3ad54f7fbaf0e9d6fadf0452, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:58,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:58,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:58,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:58,096 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:58,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:58,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:58,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:58,096 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:58,097 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,097 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:58,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:58,098 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,099 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#271 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:58,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:58,100 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:58,102 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:58,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:58,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:58,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:58,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:58,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:58,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:58,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,105 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:58,105 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:58,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:58,109 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/c2b7084debaf4c11b975f4fff709c92c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c2b7084debaf4c11b975f4fff709c92c 2023-04-13 20:54:58,113 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into c2b7084debaf4c11b975f4fff709c92c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:58,114 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:58,114 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=13, startTime=1681419298088; duration=0sec 2023-04-13 20:54:58,114 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:54:58,114 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:54:58,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:54:58,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:54:58,115 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,115 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/60abbd41caf0477baff17d40276d0f2d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.1 K 2023-04-13 20:54:58,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 60abbd41caf0477baff17d40276d0f2d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=524, earliestPutTs=1681419295412 2023-04-13 20:54:58,115 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 338ad10df3c64568b5922bc60284f1fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=561, earliestPutTs=1681419295509 2023-04-13 20:54:58,116 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3ec36c3185a84cfbaac4b762e342aec5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:54:58,122 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:54:58,205 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:58,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:58,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:58,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:58,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:58,208 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:58,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:58,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:58,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:58,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:58,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:58,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:58,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:58,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:58,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:58,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:58,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:58,211 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:58,211 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:58,212 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:58,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:54:58,219 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/7c5c26567bfe4ecd8eb10837def0c516 2023-04-13 20:54:58,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:58,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:58,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419358284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419358285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419358286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419358288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419358288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419358388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419358389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419358391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419358392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419358392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,510 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/998a3afe8f014113bc6fc855a0cb353a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/998a3afe8f014113bc6fc855a0cb353a 2023-04-13 20:54:58,513 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into 998a3afe8f014113bc6fc855a0cb353a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:58,514 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:58,514 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=13, startTime=1681419298088; duration=0sec 2023-04-13 20:54:58,514 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:58,529 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/037cf5b41103455d8c0a8612d55e20f0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/037cf5b41103455d8c0a8612d55e20f0 2023-04-13 20:54:58,533 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 037cf5b41103455d8c0a8612d55e20f0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:54:58,533 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:58,533 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=13, startTime=1681419298088; duration=0sec 2023-04-13 20:54:58,533 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:54:58,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419358592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419358592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419358593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419358594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419358596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,630 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/4e011056587245e99d05863464163cdf 2023-04-13 20:54:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419358894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419358896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419358897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419358898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:58,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:58,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419358901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,041 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=612 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/89477fc1c78f4413b7f3abd904a25a08 2023-04-13 20:54:59,048 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/7c5c26567bfe4ecd8eb10837def0c516 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516 2023-04-13 20:54:59,052 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516, entries=150, sequenceid=612, filesize=11.9 K 2023-04-13 20:54:59,053 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/4e011056587245e99d05863464163cdf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf 2023-04-13 20:54:59,056 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf, entries=150, sequenceid=612, filesize=11.9 K 2023-04-13 20:54:59,057 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/89477fc1c78f4413b7f3abd904a25a08 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08 2023-04-13 20:54:59,060 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08, entries=150, sequenceid=612, filesize=11.9 K 2023-04-13 20:54:59,061 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=46.96 KB/48090 for 5c92cecd6957c182fb4d0c5ce319c122 in 850ms, sequenceid=612, compaction requested=false 2023-04-13 20:54:59,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:54:59,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:54:59,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:59,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77280c27[Count = 0] remaining members to acquire global barrier 2023-04-13 20:54:59,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:54:59,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,067 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:54:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:54:59,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:54:59,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:54:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:54:59,068 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:59,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:59,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:54:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:54:59,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:54:59,071 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:54:59,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:54:59,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:54:59,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:54:59,072 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:54:59,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:54:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:54:59,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:59,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:59,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:54:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:54:59,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:54:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:59,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:54:59,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:54:59,079 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:54:59,079 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:54:59,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:54:59,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:54:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:54:59,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:54:59,179 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:54:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:54:59,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:54:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:54:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:54:59,182 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:54:59,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:59,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:54:59,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:54:59,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:59,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:54:59,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:54:59,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:54:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:59,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:54:59,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:54:59,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:54:59,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:54:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:54:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:54:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:54:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:54:59,185 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:54:59,185 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:54:59,185 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:54:59,186 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:54:59,186 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:54:59,196 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/decd5d8fb608496a88dd2ecfa365d3fd 2023-04-13 20:54:59,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:54:59,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:54:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419299408, max timestamp=1681419299412Num uniques 859; 2023-04-13 20:54:59,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419299408, max timestamp=1681419299412Num uniques 859; 2023-04-13 20:54:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419299408, max timestamp=1681419299412Num uniques 859; 2023-04-13 20:54:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:59,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:54:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419359426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419359426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419359427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681419299424, max timestamp=1681419299430Num uniques 1018; 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419299424, max timestamp=1681419299430Num uniques 1018; 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1681419299424, max timestamp=1681419299430Num uniques 1018; 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:54:59,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-13 20:54:59,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419359454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419359457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419359531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419359531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419359533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419359558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419359560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,606 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/fd17154ccfde42b7b7094adb5b63fbe2 2023-04-13 20:54:59,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419359733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419359734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419359736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419359761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:54:59,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:54:59,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419359764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,017 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/434cf000bca74def936e140e76448a8d 2023-04-13 20:55:00,021 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/decd5d8fb608496a88dd2ecfa365d3fd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd 2023-04-13 20:55:00,025 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd, entries=150, sequenceid=624, filesize=11.9 K 2023-04-13 20:55:00,026 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/fd17154ccfde42b7b7094adb5b63fbe2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2 2023-04-13 20:55:00,029 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2, entries=150, sequenceid=624, filesize=11.9 K 2023-04-13 20:55:00,030 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/434cf000bca74def936e140e76448a8d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d 2023-04-13 20:55:00,033 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d, entries=150, sequenceid=624, filesize=11.9 K 2023-04-13 20:55:00,034 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for 5c92cecd6957c182fb4d0c5ce319c122 in 848ms, sequenceid=624, compaction requested=true 2023-04-13 20:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:00,034 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46067da4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:00,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=143.13 KB heapSize=404.45 KB 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:00,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:00,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,045 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,045 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,047 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/0723058fbc934915ae0c5ed3ef804dcf 2023-04-13 20:55:00,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419360045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419360046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419360049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,052 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,053 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,053 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419360066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419360068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419360150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419360150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,153 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419360152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,156 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,157 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,158 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e327b01[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,169 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,173 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,173 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:00,182 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,182 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,282 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,285 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,287 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,288 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,289 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,289 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,289 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,291 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f0d156b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,295 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,297 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,298 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,304 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,304 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419360352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419360353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419360356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,404 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,408 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,413 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,414 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68736399[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,421 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,421 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,427 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,427 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/86c266cf5ab6467abbdda986edf666de 2023-04-13 20:55:00,527 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,530 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,530 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,533 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,533 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,534 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,534 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dedd86e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,539 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,540 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,542 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,542 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,551 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,551 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419360570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419360571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,652 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,655 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419360656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419360656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,658 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419360658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,663 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ea42a1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,667 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,671 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,671 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,672 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,672 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,673 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,673 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,674 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,678 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,678 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,778 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,781 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,783 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,785 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,785 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,785 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:00,785 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:00,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:00,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:00,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477bb79a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:00,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:00,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:00,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:00,791 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:00,793 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:00,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:00,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:00,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:00,793 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:00,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:00,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:00,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,798 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:00,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:00,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:00,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:00,800 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:00,800 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:00,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:00,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/509a0f17e1324e7b93351f06ff4161bc 2023-04-13 20:55:00,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/0723058fbc934915ae0c5ed3ef804dcf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf 2023-04-13 20:55:00,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf, entries=150, sequenceid=678, filesize=11.9 K 2023-04-13 20:55:00,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/86c266cf5ab6467abbdda986edf666de as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de 2023-04-13 20:55:00,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de, entries=150, sequenceid=678, filesize=11.9 K 2023-04-13 20:55:00,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/509a0f17e1324e7b93351f06ff4161bc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc 2023-04-13 20:55:00,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc, entries=150, sequenceid=678, filesize=11.9 K 2023-04-13 20:55:00,883 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~143.13 KB/146560, heapSize ~404.41 KB/414112, currentSize=40.25 KB/41220 for 5c92cecd6957c182fb4d0c5ce319c122 in 844ms, sequenceid=678, compaction requested=true 2023-04-13 20:55:00,883 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:00,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:00,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:00,883 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:00,883 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:00,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:00,885 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:55:00,885 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/998a3afe8f014113bc6fc855a0cb353a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=48.2 K 2023-04-13 20:55:00,885 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,885 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c2b7084debaf4c11b975f4fff709c92c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=48.2 K 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 998a3afe8f014113bc6fc855a0cb353a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:55:00,885 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c2b7084debaf4c11b975f4fff709c92c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:55:00,886 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7c5c26567bfe4ecd8eb10837def0c516, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681419297669 2023-04-13 20:55:00,886 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4e011056587245e99d05863464163cdf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681419297669 2023-04-13 20:55:00,886 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fd17154ccfde42b7b7094adb5b63fbe2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681419298283 2023-04-13 20:55:00,886 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting decd5d8fb608496a88dd2ecfa365d3fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681419298283 2023-04-13 20:55:00,887 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 86c266cf5ab6467abbdda986edf666de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:00,887 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0723058fbc934915ae0c5ed3ef804dcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:00,895 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:00,897 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#284 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:00,908 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:00,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:00,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:00,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:00,915 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:00,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:00,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:00,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:00,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:00,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:00,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:00,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:00,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:00,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:00,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:00,920 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:00,920 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:00,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/d7fe0feb3e7746b9bc6f71211679b030 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d7fe0feb3e7746b9bc6f71211679b030 2023-04-13 20:55:00,926 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into d7fe0feb3e7746b9bc6f71211679b030(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:00,926 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:00,926 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=12, startTime=1681419300883; duration=0sec 2023-04-13 20:55:00,926 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:00,926 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:00,927 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:00,927 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:55:00,927 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:00,927 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/037cf5b41103455d8c0a8612d55e20f0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=48.1 K 2023-04-13 20:55:00,928 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 037cf5b41103455d8c0a8612d55e20f0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=575, earliestPutTs=1681419297632 2023-04-13 20:55:00,928 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 89477fc1c78f4413b7f3abd904a25a08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=612, earliestPutTs=1681419297669 2023-04-13 20:55:00,928 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 434cf000bca74def936e140e76448a8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1681419298283 2023-04-13 20:55:00,929 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 509a0f17e1324e7b93351f06ff4161bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:00,936 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#286 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:01,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:01,161 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:01,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419301179, max timestamp=1681419301183Num uniques 859; 2023-04-13 20:55:01,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419301179, max timestamp=1681419301183Num uniques 859; 2023-04-13 20:55:01,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:01,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419301179, max timestamp=1681419301183Num uniques 859; 2023-04-13 20:55:01,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:01,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:01,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419361204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419361205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419361207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,306 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c75a1aa28003426e868dc8232e40f5b4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c75a1aa28003426e868dc8232e40f5b4 2023-04-13 20:55:01,310 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into c75a1aa28003426e868dc8232e40f5b4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:01,311 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:01,311 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=12, startTime=1681419300883; duration=0sec 2023-04-13 20:55:01,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,311 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419361308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419361308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419361311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,330 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ce66c85b0fa14e7e8ae31a81bdeab708 2023-04-13 20:55:01,340 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/1f08714238c6412a858309b12e596f2a 2023-04-13 20:55:01,345 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/1288a8099b0046e9a6d27a81af110c18 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1288a8099b0046e9a6d27a81af110c18 2023-04-13 20:55:01,350 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 1288a8099b0046e9a6d27a81af110c18(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:01,350 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:01,350 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=12, startTime=1681419300883; duration=0sec 2023-04-13 20:55:01,350 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:01,351 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/dd7c572d8c364a71b4f82aded1da7b90 2023-04-13 20:55:01,354 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ce66c85b0fa14e7e8ae31a81bdeab708 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708 2023-04-13 20:55:01,358 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708, entries=100, sequenceid=687, filesize=9.5 K 2023-04-13 20:55:01,359 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/1f08714238c6412a858309b12e596f2a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a 2023-04-13 20:55:01,362 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a, entries=100, sequenceid=687, filesize=9.5 K 2023-04-13 20:55:01,362 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/dd7c572d8c364a71b4f82aded1da7b90 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90 2023-04-13 20:55:01,365 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90, entries=100, sequenceid=687, filesize=9.5 K 2023-04-13 20:55:01,366 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 5c92cecd6957c182fb4d0c5ce319c122 in 446ms, sequenceid=687, compaction requested=false 2023-04-13 20:55:01,366 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:01,366 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:01,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:01,368 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,368 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:01,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e76b47[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:01,370 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:01,370 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,371 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:01,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:01,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:01,373 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,373 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:01,375 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:01,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:01,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:01,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:01,375 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:01,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:01,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,383 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,383 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,384 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:01,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,384 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:01,384 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:01,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,484 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:01,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:01,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:01,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:01,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:01,487 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:01,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:01,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:01,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:01,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,489 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:01,489 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:01,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:01,491 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:01,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:01,491 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:01,491 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:01,491 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 20:55:01,491 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:01,492 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:01,492 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:01,492 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:01,492 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:01,492 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:01,504 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/96e986830b4c413e8d8de758d196800b 2023-04-13 20:55:01,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:01,514 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:01,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419361518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419361519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419361520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419361574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419361579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419361620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419361621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419361622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419361822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,825 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419361823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:01,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419361825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,915 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/ac7a91a285284a7cb1479619bb74588a 2023-04-13 20:55:01,925 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=727 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/033ed26013be48a1b07e5d3b71de80f7 2023-04-13 20:55:01,929 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/96e986830b4c413e8d8de758d196800b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b 2023-04-13 20:55:01,932 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b, entries=150, sequenceid=727, filesize=11.9 K 2023-04-13 20:55:01,933 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/ac7a91a285284a7cb1479619bb74588a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a 2023-04-13 20:55:01,937 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a, entries=150, sequenceid=727, filesize=11.9 K 2023-04-13 20:55:01,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/033ed26013be48a1b07e5d3b71de80f7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7 2023-04-13 20:55:01,941 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7, entries=150, sequenceid=727, filesize=11.9 K 2023-04-13 20:55:01,942 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 5c92cecd6957c182fb4d0c5ce319c122 in 451ms, sequenceid=727, compaction requested=true 2023-04-13 20:55:01,942 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:01,942 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:01,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:01,944 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:01,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47baeb1d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:01,946 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:01,946 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,948 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:01,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:01,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:01,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:01,949 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:01,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:01,952 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:01,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:01,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:01,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:01,952 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:01,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:01,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:01,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:01,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:01,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:01,964 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:01,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:01,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:01,965 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:01,965 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:01,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:02,065 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:02,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:02,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:02,068 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:02,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:02,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:02,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,070 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:02,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:02,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:02,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:02,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:02,073 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:02,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:02,081 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ff342b243d224797bc3ff6f5cbbed1bf 2023-04-13 20:55:02,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:02,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302137, max timestamp=1681419302144Num uniques 859; 2023-04-13 20:55:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302137, max timestamp=1681419302144Num uniques 859; 2023-04-13 20:55:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302137, max timestamp=1681419302144Num uniques 859; 2023-04-13 20:55:02,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,166 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419302158, max timestamp=1681419302164Num uniques 1018; 2023-04-13 20:55:02,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419362163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:02,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419302158, max timestamp=1681419302164Num uniques 1018; 2023-04-13 20:55:02,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-13 20:55:02,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419302158, max timestamp=1681419302164Num uniques 1018; 2023-04-13 20:55:02,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:02,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419362197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,199 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419302183, max timestamp=1681419302197Num uniques 1077; 2023-04-13 20:55:02,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419302183, max timestamp=1681419302197Num uniques 1027; 2023-04-13 20:55:02,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=10392, suffix heap size=139976,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=227 2023-04-13 20:55:02,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419302183, max timestamp=1681419302197Num uniques 1027; 2023-04-13 20:55:02,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:55:02,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:55:02,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419362260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419362267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419362300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419362364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419362470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,491 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/5e52c8a562d743a3af9fd28295e7cbf7 2023-04-13 20:55:02,500 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=736 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/830d2f5e3605407fa7af976df7834001 2023-04-13 20:55:02,504 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/ff342b243d224797bc3ff6f5cbbed1bf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf 2023-04-13 20:55:02,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419362504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,507 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf, entries=100, sequenceid=736, filesize=9.5 K 2023-04-13 20:55:02,508 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/5e52c8a562d743a3af9fd28295e7cbf7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7 2023-04-13 20:55:02,513 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7, entries=100, sequenceid=736, filesize=9.5 K 2023-04-13 20:55:02,514 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/830d2f5e3605407fa7af976df7834001 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001 2023-04-13 20:55:02,519 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001, entries=100, sequenceid=736, filesize=9.5 K 2023-04-13 20:55:02,519 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 5c92cecd6957c182fb4d0c5ce319c122 in 446ms, sequenceid=736, compaction requested=true 2023-04-13 20:55:02,520 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:02,520 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:02,520 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:02,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e75601[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:02,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:02,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,525 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:02,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:02,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,526 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:02,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:02,529 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:02,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:02,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:02,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:02,529 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:02,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:02,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,546 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:02,546 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:02,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:02,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302225, max timestamp=1681419302567Num uniques 1086; 2023-04-13 20:55:02,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:55:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302225, max timestamp=1681419302567Num uniques 1086; 2023-04-13 20:55:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:55:02,570 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=107.34 KB heapSize=300.67 KB 2023-04-13 20:55:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302225, max timestamp=1681419302567Num uniques 1086; 2023-04-13 20:55:02,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:55:02,570 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:02,570 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:02,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:02,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:02,571 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:02,571 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:02,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/e9a24288b4a94dfcaee4102e02624162 2023-04-13 20:55:02,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/8abdcb18d2e74c09b5fbdc457c92cf46 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302628, max timestamp=1681419302637Num uniques 859; 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302628, max timestamp=1681419302637Num uniques 859; 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419302628, max timestamp=1681419302637Num uniques 859; 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:02,646 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:02,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:02,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:02,649 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:02,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:02,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:02,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,651 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:02,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:02,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:02,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:02,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:02,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:02,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:02,653 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:02,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,655 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:02,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3aea40[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:02,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:02,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,658 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:02,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,659 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:02,662 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:02,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:02,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:02,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:02,662 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:02,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,663 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,664 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:02,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,668 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:02,668 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419302676, max timestamp=1681419302691Num uniques 1018; 2023-04-13 20:55:02,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:02,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419302676, max timestamp=1681419302691Num uniques 1018; 2023-04-13 20:55:02,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419302676, max timestamp=1681419302691Num uniques 1018; 2023-04-13 20:55:02,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:02,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:02,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419302721, max timestamp=1681419302750Num uniques 1027; 2023-04-13 20:55:02,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:55:02,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419302721, max timestamp=1681419302750Num uniques 1027; 2023-04-13 20:55:02,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419302721, max timestamp=1681419302750Num uniques 1027; 2023-04-13 20:55:02,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:55:02,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:55:02,768 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:02,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:02,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:02,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:02,771 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:02,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:02,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:02,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,772 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,774 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:02,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:02,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:02,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:02,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:02,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:02,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:02,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:02,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:02,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70881bb3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:02,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:02,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:02,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:02,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:02,783 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:02,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:02,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:02,785 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:02,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:02,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:02,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:02,786 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:02,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:02,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:02,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:02,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,795 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:02,795 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:02,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302787, max timestamp=1681419302805Num uniques 1036; 2023-04-13 20:55:02,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 20:55:02,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302787, max timestamp=1681419302805Num uniques 1036; 2023-04-13 20:55:02,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 20:55:02,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419302787, max timestamp=1681419302805Num uniques 1036; 2023-04-13 20:55:02,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-13 20:55:02,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681419302821, max timestamp=1681419302829Num uniques 1045; 2023-04-13 20:55:02,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-13 20:55:02,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681419302821, max timestamp=1681419302829Num uniques 1045; 2023-04-13 20:55:02,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-13 20:55:02,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1681419302821, max timestamp=1681419302829Num uniques 1045; 2023-04-13 20:55:02,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=13510, suffix heap size=136648,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=295 2023-04-13 20:55:02,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:02,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419362848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419302845, max timestamp=1681419302849Num uniques 1154; 2023-04-13 20:55:02,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-13 20:55:02,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419302845, max timestamp=1681419302849Num uniques 1154; 2023-04-13 20:55:02,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-13 20:55:02,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1681419302845, max timestamp=1681419302849Num uniques 1154; 2023-04-13 20:55:02,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52850, new segment data size=9340, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-04-13 20:55:02,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419302872, max timestamp=1681419302877Num uniques 1063; 2023-04-13 20:55:02,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419302872, max timestamp=1681419302877Num uniques 1063; 2023-04-13 20:55:02,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 20:55:02,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 20:55:02,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419302872, max timestamp=1681419302877Num uniques 1063; 2023-04-13 20:55:02,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-13 20:55:02,895 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:02,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:02,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:02,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:02,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:02,898 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:02,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:02,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:02,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:02,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:02,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:02,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:02,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:02,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,901 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:02,902 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:02,902 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:02,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:02,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:02,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60fae4bc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:02,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:02,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:02,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:02,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,909 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:02,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419302900, max timestamp=1681419302909Num uniques 1022; 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 20:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419302900, max timestamp=1681419302909Num uniques 1022; 2023-04-13 20:55:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419302900, max timestamp=1681419302909Num uniques 1022; 2023-04-13 20:55:02,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-04-13 20:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:02,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:02,913 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:02,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:02,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:02,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:02,914 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:02,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:02,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:02,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:02,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,917 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:02,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:02,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:02,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:02,920 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:02,920 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:02,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:02,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:02,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419302927, max timestamp=1681419302942Num uniques 1081; 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419302927, max timestamp=1681419302942Num uniques 1081; 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419302927, max timestamp=1681419302942Num uniques 1081; 2023-04-13 20:55:02,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419302959, max timestamp=1681419302968Num uniques 1040; 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419302959, max timestamp=1681419302968Num uniques 1040; 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1681419302959, max timestamp=1681419302968Num uniques 1040; 2023-04-13 20:55:02,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681419302979, max timestamp=1681419302993Num uniques 1099; 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9107, totalHeapSize=20936, min timestamp=1681419302979, max timestamp=1681419302993Num uniques 1049; 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1681419302979, max timestamp=1681419302993Num uniques 1049; 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=9104, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48034, new segment data size=9104, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-13 20:55:02,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48037, new segment data size=9107, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-13 20:55:03,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e94397f8a6b944b8b845128e875a85d8 2023-04-13 20:55:03,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/e9a24288b4a94dfcaee4102e02624162 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162 2023-04-13 20:55:03,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162, entries=150, sequenceid=808, filesize=11.9 K 2023-04-13 20:55:03,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/8abdcb18d2e74c09b5fbdc457c92cf46 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46 2023-04-13 20:55:03,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46, entries=150, sequenceid=808, filesize=11.9 K 2023-04-13 20:55:03,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e94397f8a6b944b8b845128e875a85d8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8 2023-04-13 20:55:03,020 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,023 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8, entries=150, sequenceid=808, filesize=11.9 K 2023-04-13 20:55:03,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f72e1fc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,035 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,035 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,037 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.66 KB/65184, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 467ms, sequenceid=808, compaction requested=true 2023-04-13 20:55:03,037 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:03,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:03,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:03,037 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:55:03,037 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:55:03,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:03,037 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:03,038 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=100.63 KB heapSize=281.63 KB 2023-04-13 20:55:03,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:03,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:03,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,038 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:03,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,038 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,039 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:03,039 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56758 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,039 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,039 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56758 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:55:03,039 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,039 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:55:03,039 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d7fe0feb3e7746b9bc6f71211679b030, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=55.4 K 2023-04-13 20:55:03,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,039 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,040 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c75a1aa28003426e868dc8232e40f5b4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=55.4 K 2023-04-13 20:55:03,040 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d7fe0feb3e7746b9bc6f71211679b030, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:03,040 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c75a1aa28003426e868dc8232e40f5b4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:03,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,040 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1f08714238c6412a858309b12e596f2a, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681419300044 2023-04-13 20:55:03,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,041 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ce66c85b0fa14e7e8ae31a81bdeab708, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681419300044 2023-04-13 20:55:03,041 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ac7a91a285284a7cb1479619bb74588a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681419301200 2023-04-13 20:55:03,041 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 96e986830b4c413e8d8de758d196800b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681419301200 2023-04-13 20:55:03,041 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5e52c8a562d743a3af9fd28295e7cbf7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681419301518 2023-04-13 20:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,042 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8abdcb18d2e74c09b5fbdc457c92cf46, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:03,042 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ff342b243d224797bc3ff6f5cbbed1bf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681419301518 2023-04-13 20:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,043 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e9a24288b4a94dfcaee4102e02624162, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:03,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,044 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,044 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:03,058 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:03,059 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#299 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:03,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419363063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419363064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419363065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1011 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c8cd308ff73a45c592b05c8e53055e51 2023-04-13 20:55:03,144 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,147 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,153 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@173274b8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,161 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,164 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,164 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,166 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419363167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,173 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,173 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419363170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419363170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,273 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,276 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,280 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27d7b06e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,287 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,295 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:03,304 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,304 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,304 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419363369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419363374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419363374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,404 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,407 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,411 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,411 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,412 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,414 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10bf6f2c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,421 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,421 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,421 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,433 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,433 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,482 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/a1acfdc7acef4397a8b9704446bd4249 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a1acfdc7acef4397a8b9704446bd4249 2023-04-13 20:55:03,482 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/18e38a07d1784f798fdb8d734ee862a2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/18e38a07d1784f798fdb8d734ee862a2 2023-04-13 20:55:03,486 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into a1acfdc7acef4397a8b9704446bd4249(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:03,486 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into 18e38a07d1784f798fdb8d734ee862a2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:03,487 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:03,487 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:03,487 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=11, startTime=1681419303037; duration=0sec 2023-04-13 20:55:03,487 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=11, startTime=1681419303037; duration=0sec 2023-04-13 20:55:03,487 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:03,487 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:03,487 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:55:03,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1011 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/c1c0338dddac4366bea7528d2a849604 2023-04-13 20:55:03,489 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56724 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:55:03,489 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:55:03,490 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,490 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1288a8099b0046e9a6d27a81af110c18, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=55.4 K 2023-04-13 20:55:03,490 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1288a8099b0046e9a6d27a81af110c18, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1681419299447 2023-04-13 20:55:03,491 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting dd7c572d8c364a71b4f82aded1da7b90, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1681419300044 2023-04-13 20:55:03,491 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 033ed26013be48a1b07e5d3b71de80f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=727, earliestPutTs=1681419301200 2023-04-13 20:55:03,491 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 830d2f5e3605407fa7af976df7834001, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=736, earliestPutTs=1681419301518 2023-04-13 20:55:03,492 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e94397f8a6b944b8b845128e875a85d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:03,510 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#303 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:03,533 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,556 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1011 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/929afd1649e14faaa91e39457d868213 2023-04-13 20:55:03,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,556 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/54944c59cca048e4aaa5c0abae0dfe05 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/54944c59cca048e4aaa5c0abae0dfe05 2023-04-13 20:55:03,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,563 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 54944c59cca048e4aaa5c0abae0dfe05(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:03,563 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:03,563 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=11, startTime=1681419303037; duration=0sec 2023-04-13 20:55:03,564 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:03,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/c8cd308ff73a45c592b05c8e53055e51 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,568 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,568 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,568 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,568 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51, entries=250, sequenceid=1011, filesize=16.7 K 2023-04-13 20:55:03,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/c1c0338dddac4366bea7528d2a849604 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604 2023-04-13 20:55:03,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38ea01c6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,573 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604, entries=150, sequenceid=1011, filesize=11.9 K 2023-04-13 20:55:03,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,574 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/929afd1649e14faaa91e39457d868213 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213 2023-04-13 20:55:03,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213, entries=150, sequenceid=1011, filesize=11.9 K 2023-04-13 20:55:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~281.58 KB/288336, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 543ms, sequenceid=1011, compaction requested=false 2023-04-13 20:55:03,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:03,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,581 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,582 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:03,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:03,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,590 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,593 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,594 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,594 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:03,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/448b3f43faf74eb0bdc35f55d17de522 2023-04-13 20:55:03,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419363615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,618 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419363616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,620 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/a09c7143e72341f79d6e728307341682 2023-04-13 20:55:03,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419363672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419363677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419363679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,694 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,696 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,700 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,700 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7935d7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,710 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,710 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,717 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,717 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419363718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419363719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,817 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,820 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,820 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,821 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,821 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,823 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,823 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,823 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,823 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,824 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b7b6f6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,828 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,831 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,831 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,831 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,833 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,833 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,839 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,839 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419363923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:03,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419363923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,939 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:03,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:03,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:03,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:03,942 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:03,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:03,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:03,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,944 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,944 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,944 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,944 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:03,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:03,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:03,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:03,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:03,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:03,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:03,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:03,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:03,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,953 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:03,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:03,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d1a7b03[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:03,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,956 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:03,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:03,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:03,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,957 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:03,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:03,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:03,960 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:03,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:03,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:03,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:03,960 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:03,961 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:03,961 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,961 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:03,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,963 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:03,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:03,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:03,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:03,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:03,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:03,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:03,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:03,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:03,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:03,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:03,970 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:03,970 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:03,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:03,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:03,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:03,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1030 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e097a64e996043debd66c524252b500a 2023-04-13 20:55:04,033 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/448b3f43faf74eb0bdc35f55d17de522 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522 2023-04-13 20:55:04,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522, entries=150, sequenceid=1030, filesize=11.9 K 2023-04-13 20:55:04,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/a09c7143e72341f79d6e728307341682 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682 2023-04-13 20:55:04,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682, entries=150, sequenceid=1030, filesize=11.9 K 2023-04-13 20:55:04,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/e097a64e996043debd66c524252b500a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a 2023-04-13 20:55:04,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a, entries=150, sequenceid=1030, filesize=11.9 K 2023-04-13 20:55:04,048 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5c92cecd6957c182fb4d0c5ce319c122 in 459ms, sequenceid=1030, compaction requested=true 2023-04-13 20:55:04,048 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:04,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:04,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:04,049 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:04,049 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:04,049 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:04,050 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42324 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:04,050 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:04,050 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:55:04,050 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:55:04,050 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,050 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,050 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/18e38a07d1784f798fdb8d734ee862a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=41.3 K 2023-04-13 20:55:04,050 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a1acfdc7acef4397a8b9704446bd4249, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.6 K 2023-04-13 20:55:04,050 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 18e38a07d1784f798fdb8d734ee862a2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:04,051 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a1acfdc7acef4397a8b9704446bd4249, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:04,051 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c1c0338dddac4366bea7528d2a849604, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1011, earliestPutTs=1681419303002 2023-04-13 20:55:04,051 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c8cd308ff73a45c592b05c8e53055e51, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1011, earliestPutTs=1681419303002 2023-04-13 20:55:04,051 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a09c7143e72341f79d6e728307341682, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:04,051 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 448b3f43faf74eb0bdc35f55d17de522, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:04,058 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:04,058 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#307 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:04,070 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:04,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:04,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:04,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:04,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:04,074 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:04,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:04,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:04,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,075 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,077 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,077 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:04,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:04,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:04,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:04,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:04,078 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,078 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:04,078 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:04,079 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:04,181 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:04,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419364218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419364218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419364218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419364225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419364227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419364321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419364321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419364321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,469 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/35f23a6a53da48a2b933f479c0be9c2a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/35f23a6a53da48a2b933f479c0be9c2a 2023-04-13 20:55:04,473 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into 35f23a6a53da48a2b933f479c0be9c2a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:04,474 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:04,474 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=13, startTime=1681419304049; duration=0sec 2023-04-13 20:55:04,474 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:04,474 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:04,475 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:04,475 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:55:04,475 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,475 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/54944c59cca048e4aaa5c0abae0dfe05, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.5 K 2023-04-13 20:55:04,476 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 54944c59cca048e4aaa5c0abae0dfe05, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1681419302225 2023-04-13 20:55:04,476 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 929afd1649e14faaa91e39457d868213, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1011, earliestPutTs=1681419303002 2023-04-13 20:55:04,476 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e097a64e996043debd66c524252b500a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:04,478 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/fc902f853c3347d7b8b62f1635898c9a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fc902f853c3347d7b8b62f1635898c9a 2023-04-13 20:55:04,482 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into fc902f853c3347d7b8b62f1635898c9a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:04,482 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:04,483 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=13, startTime=1681419304049; duration=0sec 2023-04-13 20:55:04,483 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:04,483 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#310 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:04,489 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/14a8f18b3671480f81bfb082f0588e08 2023-04-13 20:55:04,498 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/d11b9e9338b144688a68d34869895793 2023-04-13 20:55:04,507 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5c971ec7d2dd4d459132f9968e95296b 2023-04-13 20:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/14a8f18b3671480f81bfb082f0588e08 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08 2023-04-13 20:55:04,514 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08, entries=150, sequenceid=1046, filesize=11.9 K 2023-04-13 20:55:04,515 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/d11b9e9338b144688a68d34869895793 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793 2023-04-13 20:55:04,518 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793, entries=150, sequenceid=1046, filesize=11.9 K 2023-04-13 20:55:04,519 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5c971ec7d2dd4d459132f9968e95296b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b 2023-04-13 20:55:04,522 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b, entries=150, sequenceid=1046, filesize=11.9 K 2023-04-13 20:55:04,523 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=100.63 KB/103050 for 5c92cecd6957c182fb4d0c5ce319c122 in 445ms, sequenceid=1046, compaction requested=false 2023-04-13 20:55:04,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:04,523 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:04,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:04,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:04,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:04,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,525 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,525 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:04,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:04,526 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:04,526 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:04,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:04,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2516a757[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:04,528 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:04,528 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,531 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:04,531 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:04,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:04,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/b962fd52ecf64cc3adc2aa3e595585f0 2023-04-13 20:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:04,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:04,538 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:04,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:04,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:04,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:04,538 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:04,539 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,539 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419364536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,539 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419364537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419364539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:04,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:04,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,544 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:04,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,544 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:04,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,651 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:04,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419364641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:04,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419364641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:04,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419364652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:04,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:04,655 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:04,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:04,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:04,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,658 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:04,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:04,659 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:04,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:04,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:04,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:04,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:04,661 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:04,663 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:04,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b4e9af[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:04,663 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,664 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:04,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:04,665 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:04,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:04,668 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:04,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:04,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:04,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:04,668 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:04,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,670 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:04,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:04,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:04,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,674 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,674 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,675 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:04,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,675 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:04,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419364730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419364731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,775 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:04,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:04,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:04,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:04,778 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:04,778 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,778 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:04,778 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:04,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:04,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:04,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:04,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:04,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:04,782 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,782 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:04,782 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:04,782 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:04,782 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:04,783 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:04,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:04,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2ff549[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:04,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,787 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:04,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:04,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:04,788 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:04,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:04,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:04,791 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:04,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:04,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:04,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:04,791 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:04,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:04,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,802 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:04,802 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:04,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:04,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,802 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,802 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419364854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419364854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:04,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419364855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,891 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/ffa9703a64324af5817d61423d239c7b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ffa9703a64324af5817d61423d239c7b 2023-04-13 20:55:04,895 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into ffa9703a64324af5817d61423d239c7b(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:04,895 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:04,895 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=13, startTime=1681419304049; duration=0sec 2023-04-13 20:55:04,895 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:04,902 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:04,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:04,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:04,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:04,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:04,904 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:04,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:04,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:04,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,908 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,909 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:04,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:04,910 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,910 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:04,910 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:04,910 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:04,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:04,911 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:04,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55735d4c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:04,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:04,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:04,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:04,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:04,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:04,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:04,919 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:04,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:04,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:04,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:04,919 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:04,921 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,921 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:04,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,921 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:04,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:04,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,923 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:04,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:04,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:04,925 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:04,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:04,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:04,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:04,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:04,925 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:04,925 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:04,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/ca175a46021446458f641b0352035c84 2023-04-13 20:55:05,025 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:05,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:05,029 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:05,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:05,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:05,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,030 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,031 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:05,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:05,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:05,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:05,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,032 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:05,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:05,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454f74aa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:05,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:05,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:05,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:05,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:05,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:05,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:05,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:05,041 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:05,042 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:05,042 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:05,042 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:05,042 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:05,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:05,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:05,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:05,047 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,047 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,048 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:05,048 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:05,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,148 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:05,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:05,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:05,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:05,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:05,151 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:05,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:05,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:05,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,154 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:05,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:05,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:05,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:05,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:05,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:05,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:05,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:05,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419365158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:05,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c1043d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:05,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:05,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419365158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419365159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,161 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:05,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:05,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:05,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:05,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:05,165 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:05,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:05,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:05,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:05,165 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:05,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:05,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:05,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,175 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:05,175 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:05,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,275 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:05,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:05,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:05,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:05,278 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:05,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:05,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:05,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,280 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:05,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:05,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:05,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:05,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:05,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:05,282 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:05,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:05,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,285 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,285 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:05,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:05,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6702a91[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:05,287 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:05,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:05,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,289 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,290 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:05,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:05,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:05,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:05,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:05,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:05,293 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:05,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:05,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:05,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:05,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:05,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:05,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:05,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:05,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:05,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:05,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:05,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:05,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,304 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:05,304 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:05,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:05,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1067 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/a3586f5dfc324e9c8751394751c457bc 2023-04-13 20:55:05,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/b962fd52ecf64cc3adc2aa3e595585f0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0 2023-04-13 20:55:05,374 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 20:55:05,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/ca175a46021446458f641b0352035c84 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84 2023-04-13 20:55:05,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 20:55:05,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/a3586f5dfc324e9c8751394751c457bc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc 2023-04-13 20:55:05,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc, entries=150, sequenceid=1067, filesize=11.9 K 2023-04-13 20:55:05,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5c92cecd6957c182fb4d0c5ce319c122 in 857ms, sequenceid=1067, compaction requested=true 2023-04-13 20:55:05,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:05,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:05,383 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:05,383 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:05,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:05,384 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:05,384 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:05,385 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:05,385 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:55:05,385 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:55:05,385 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,385 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,385 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fc902f853c3347d7b8b62f1635898c9a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.7 K 2023-04-13 20:55:05,385 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/35f23a6a53da48a2b933f479c0be9c2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.7 K 2023-04-13 20:55:05,385 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fc902f853c3347d7b8b62f1635898c9a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:05,385 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 35f23a6a53da48a2b933f479c0be9c2a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:05,386 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 14a8f18b3671480f81bfb082f0588e08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1681419303605 2023-04-13 20:55:05,386 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d11b9e9338b144688a68d34869895793, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1681419303605 2023-04-13 20:55:05,386 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b962fd52ecf64cc3adc2aa3e595585f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:05,386 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ca175a46021446458f641b0352035c84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:05,396 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:05,396 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#317 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:05,404 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:05,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:05,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:05,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:05,409 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:05,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:05,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:05,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,410 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:05,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:05,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:05,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:05,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:05,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:05,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:05,418 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/cd4c34cc10bd44adaf23a9adba1bd3da as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/cd4c34cc10bd44adaf23a9adba1bd3da 2023-04-13 20:55:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:05,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:05,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:05,420 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:55:05,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:05,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:05,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:05,421 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:05,421 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:05,421 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:05,430 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into cd4c34cc10bd44adaf23a9adba1bd3da(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:05,430 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:05,430 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=13, startTime=1681419305383; duration=0sec 2023-04-13 20:55:05,430 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:05,430 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:05,431 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:05,431 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:55:05,431 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:05,432 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ffa9703a64324af5817d61423d239c7b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=36.6 K 2023-04-13 20:55:05,432 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ffa9703a64324af5817d61423d239c7b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=1030, earliestPutTs=1681419303062 2023-04-13 20:55:05,436 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5c971ec7d2dd4d459132f9968e95296b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1681419303605 2023-04-13 20:55:05,437 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a3586f5dfc324e9c8751394751c457bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:05,457 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#319 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:05,485 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/8ca4b0bd512942abb556a2925f467473 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/8ca4b0bd512942abb556a2925f467473 2023-04-13 20:55:05,489 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 8ca4b0bd512942abb556a2925f467473(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:05,490 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:05,490 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=13, startTime=1681419305384; duration=0sec 2023-04-13 20:55:05,490 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:05,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:05,665 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:05,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419365678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419365679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419365679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419365732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419365742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419365781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419365781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419365782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,808 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/f3e608ac325346bdb2e36962495f9d91 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f3e608ac325346bdb2e36962495f9d91 2023-04-13 20:55:05,813 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into f3e608ac325346bdb2e36962495f9d91(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:05,813 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:05,813 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=13, startTime=1681419305383; duration=0sec 2023-04-13 20:55:05,813 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:05,866 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/709edc5eac994632a4dd7254a8af72cb 2023-04-13 20:55:05,877 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/6a338b9ea12845f39bc954741662748a 2023-04-13 20:55:05,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419365984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419365984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:05,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:05,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419365985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419366286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,288 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5d82ebf71361492cadc57386381b5af3 2023-04-13 20:55:06,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419366287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419366288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,292 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/709edc5eac994632a4dd7254a8af72cb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb 2023-04-13 20:55:06,295 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-13 20:55:06,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/6a338b9ea12845f39bc954741662748a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a 2023-04-13 20:55:06,299 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-13 20:55:06,300 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5d82ebf71361492cadc57386381b5af3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3 2023-04-13 20:55:06,304 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3, entries=150, sequenceid=1081, filesize=11.9 K 2023-04-13 20:55:06,305 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5c92cecd6957c182fb4d0c5ce319c122 in 885ms, sequenceid=1081, compaction requested=false 2023-04-13 20:55:06,305 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:06,305 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:06,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:06,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:06,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:06,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:06,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:06,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49a69e2e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:06,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:06,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,310 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:06,310 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:06,310 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:06,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:06,312 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:06,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:06,315 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:06,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:06,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:06,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:06,315 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:06,317 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,317 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:06,317 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:06,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,321 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:06,321 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:06,321 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:06,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:06,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:06,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:06,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:06,321 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:06,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,322 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:06,322 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:06,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,422 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:06,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:06,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:06,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:06,426 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:06,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:06,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:06,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:06,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:06,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:06,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:06,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:06,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:06,429 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:06,429 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:06,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:06,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:06,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419366799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419366801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419366802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,841 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/5db009d99be342cfb62928f9d978f6f6 2023-04-13 20:55:06,853 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/45819100d5604f0d9eaca517c6f08c1c 2023-04-13 20:55:06,862 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/fbea4d957e51491292c096e599f5d379 2023-04-13 20:55:06,865 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/5db009d99be342cfb62928f9d978f6f6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6 2023-04-13 20:55:06,869 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6, entries=150, sequenceid=1103, filesize=11.9 K 2023-04-13 20:55:06,869 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/45819100d5604f0d9eaca517c6f08c1c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c 2023-04-13 20:55:06,873 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c, entries=150, sequenceid=1103, filesize=11.9 K 2023-04-13 20:55:06,874 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/fbea4d957e51491292c096e599f5d379 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379 2023-04-13 20:55:06,877 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379, entries=150, sequenceid=1103, filesize=11.9 K 2023-04-13 20:55:06,878 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5c92cecd6957c182fb4d0c5ce319c122 in 449ms, sequenceid=1103, compaction requested=true 2023-04-13 20:55:06,878 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:06,878 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:06,878 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:06,880 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,881 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,881 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:06,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:06,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f24994[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:06,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:06,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,884 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:06,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:06,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:06,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:06,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:06,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:06,889 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:06,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:06,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:06,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:06,889 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:06,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:06,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:06,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:06,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:06,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:06,896 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:06,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:06,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:06,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:06,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:06,896 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:06,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:06,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:06,897 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:06,897 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:06,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:06,905 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:06,905 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:06,912 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1117 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/f3ad6e0945af417392ad69a52f778071 2023-04-13 20:55:06,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1117 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/180fa290e15e4e7a89c5e808fdfd16b6 2023-04-13 20:55:06,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419366923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419366925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:06,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419366925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:06,997 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:06,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:06,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:07,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:07,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:07,000 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:07,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:07,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:07,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:07,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:07,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:07,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:07,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:07,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:07,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:07,004 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:07,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:07,005 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:07,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4da84bbb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:07,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:07,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,011 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:07,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:07,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:07,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:07,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:07,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:07,015 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:07,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:07,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:07,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:07,015 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:07,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:07,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,021 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:07,021 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419367028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419367031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419367031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,121 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:07,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:07,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:07,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:07,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:07,124 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:07,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:07,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:07,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,127 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:07,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:07,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:07,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:07,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:07,128 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:07,129 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,129 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:07,129 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:07,129 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:07,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:07,130 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,130 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:07,132 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:07,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a8bd43d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:07,132 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,133 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,134 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:07,134 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:07,134 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:07,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:07,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:07,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:07,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:07,137 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:07,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:07,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:07,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:07,137 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:07,140 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,140 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,140 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,144 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:07,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:07,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,147 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:07,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,147 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:07,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:07,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419367232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419367234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419367235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,247 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:07,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:07,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:07,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:07,250 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,251 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,251 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,252 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,252 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:07,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:07,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:07,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:07,253 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,253 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:07,253 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:07,253 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:07,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:07,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,255 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:07,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:07,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@372174ac[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:07,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:07,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:07,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:07,259 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,259 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:07,260 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:07,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:07,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:07,263 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:07,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:07,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:07,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:07,263 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:07,265 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,265 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:07,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,265 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:07,265 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:07,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,267 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:07,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:07,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:07,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,269 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:07,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:07,269 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:07,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:07,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:07,333 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1117 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5d45789685f64dde9dd83be63c185538 2023-04-13 20:55:07,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/f3ad6e0945af417392ad69a52f778071 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071 2023-04-13 20:55:07,340 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071, entries=200, sequenceid=1117, filesize=14.3 K 2023-04-13 20:55:07,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/180fa290e15e4e7a89c5e808fdfd16b6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6 2023-04-13 20:55:07,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6, entries=150, sequenceid=1117, filesize=11.9 K 2023-04-13 20:55:07,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/5d45789685f64dde9dd83be63c185538 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538 2023-04-13 20:55:07,353 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538, entries=150, sequenceid=1117, filesize=11.9 K 2023-04-13 20:55:07,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 5c92cecd6957c182fb4d0c5ce319c122 in 448ms, sequenceid=1117, compaction requested=true 2023-04-13 20:55:07,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:07,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:07,354 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:07,354 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:07,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:07,354 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52274 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/A is initiating minor compaction (all files) 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:07,355 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/A in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/B is initiating minor compaction (all files) 2023-04-13 20:55:07,355 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/cd4c34cc10bd44adaf23a9adba1bd3da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=51.0 K 2023-04-13 20:55:07,355 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/B in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,355 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f3e608ac325346bdb2e36962495f9d91, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=48.7 K 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cd4c34cc10bd44adaf23a9adba1bd3da, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:07,355 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f3e608ac325346bdb2e36962495f9d91, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:07,356 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 709edc5eac994632a4dd7254a8af72cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681419304535 2023-04-13 20:55:07,356 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6a338b9ea12845f39bc954741662748a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681419304535 2023-04-13 20:55:07,356 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5db009d99be342cfb62928f9d978f6f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1681419305674 2023-04-13 20:55:07,356 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 45819100d5604f0d9eaca517c6f08c1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1681419305674 2023-04-13 20:55:07,356 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f3ad6e0945af417392ad69a52f778071, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1117, earliestPutTs=1681419306798 2023-04-13 20:55:07,357 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 180fa290e15e4e7a89c5e808fdfd16b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1117, earliestPutTs=1681419306798 2023-04-13 20:55:07,365 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#A#compaction#328 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:07,366 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#B#compaction#329 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:07,370 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:07,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:07,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:07,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:07,373 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:07,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:07,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:07,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,373 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,375 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:07,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:07,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:07,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:07,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:07,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:07,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:07,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:07,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:07,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:07,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:07,377 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:07,380 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/cd6c66fc584648339ce4d6e23c659459 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cd6c66fc584648339ce4d6e23c659459 2023-04-13 20:55:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:07,380 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:07,380 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:07,380 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:07,386 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/B of 5c92cecd6957c182fb4d0c5ce319c122 into cd6c66fc584648339ce4d6e23c659459(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:07,386 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:07,386 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/B, priority=12, startTime=1681419307354; duration=0sec 2023-04-13 20:55:07,387 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:07,387 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:07,388 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:07,388 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 5c92cecd6957c182fb4d0c5ce319c122/C is initiating minor compaction (all files) 2023-04-13 20:55:07,388 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5c92cecd6957c182fb4d0c5ce319c122/C in TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:07,388 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/8ca4b0bd512942abb556a2925f467473, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp, totalSize=48.6 K 2023-04-13 20:55:07,389 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8ca4b0bd512942abb556a2925f467473, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1067, earliestPutTs=1681419304190 2023-04-13 20:55:07,389 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5d82ebf71361492cadc57386381b5af3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1681419304535 2023-04-13 20:55:07,389 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fbea4d957e51491292c096e599f5d379, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1103, earliestPutTs=1681419305674 2023-04-13 20:55:07,390 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5d45789685f64dde9dd83be63c185538, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1117, earliestPutTs=1681419306798 2023-04-13 20:55:07,390 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1135 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/5d78d17845d64ce1a5020cc5e375ff61 2023-04-13 20:55:07,397 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 5c92cecd6957c182fb4d0c5ce319c122#C#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:07,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:07,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:07,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419367550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419367551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 429 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419367551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419367654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419367654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419367654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48272 deadline: 1681419367748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,752 DEBUG [Thread-1798] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:07,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48278 deadline: 1681419367749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,754 DEBUG [Thread-1796] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:07,779 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/2607d5569c7541c08116373916df6d01 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/2607d5569c7541c08116373916df6d01 2023-04-13 20:55:07,783 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/A of 5c92cecd6957c182fb4d0c5ce319c122 into 2607d5569c7541c08116373916df6d01(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:07,783 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:07,783 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/A, priority=12, startTime=1681419307354; duration=0sec 2023-04-13 20:55:07,783 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:07,809 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/74252e34889d4f20970988b7154da488 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/74252e34889d4f20970988b7154da488 2023-04-13 20:55:07,813 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5c92cecd6957c182fb4d0c5ce319c122/C of 5c92cecd6957c182fb4d0c5ce319c122 into 74252e34889d4f20970988b7154da488(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:07,813 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:07,814 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122., storeName=5c92cecd6957c182fb4d0c5ce319c122/C, priority=12, startTime=1681419307354; duration=0sec 2023-04-13 20:55:07,814 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:07,814 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1135 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/502e9584ffff469396b2891959fc3402 2023-04-13 20:55:07,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419367856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419367856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:07,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:07,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419367858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419368159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419368159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419368160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,243 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1135 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/da85e5dbb9b1484f8b3e5d8a4cd65fca 2023-04-13 20:55:08,247 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/5d78d17845d64ce1a5020cc5e375ff61 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5d78d17845d64ce1a5020cc5e375ff61 2023-04-13 20:55:08,250 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5d78d17845d64ce1a5020cc5e375ff61, entries=150, sequenceid=1135, filesize=11.9 K 2023-04-13 20:55:08,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/502e9584ffff469396b2891959fc3402 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/502e9584ffff469396b2891959fc3402 2023-04-13 20:55:08,254 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/502e9584ffff469396b2891959fc3402, entries=150, sequenceid=1135, filesize=11.9 K 2023-04-13 20:55:08,255 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/da85e5dbb9b1484f8b3e5d8a4cd65fca as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/da85e5dbb9b1484f8b3e5d8a4cd65fca 2023-04-13 20:55:08,258 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/da85e5dbb9b1484f8b3e5d8a4cd65fca, entries=150, sequenceid=1135, filesize=11.9 K 2023-04-13 20:55:08,258 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 878ms, sequenceid=1135, compaction requested=false 2023-04-13 20:55:08,258 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:08,258 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:08,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:08,260 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,260 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:08,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@527a2535[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:08,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:08,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,263 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:08,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,264 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:08,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:08,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:08,268 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:08,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:08,271 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:08,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:08,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:08,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:08,271 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:08,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:08,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:08,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,274 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,278 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:08,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:08,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:08,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:08,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:08,278 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:08,278 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:08,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:08,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:08,378 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:08,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:08,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:08,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:08,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:08,381 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:08,381 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:08,381 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:08,381 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:08,382 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,382 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,383 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:08,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:08,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:08,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:08,384 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:08,384 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:08,384 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:08,385 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:08,385 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:08,385 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:08,385 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:08,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:08,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419368675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419368675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419368676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 444 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419368778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419368778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419368779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,795 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1153 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/25b0c4e062994564890b6f3a4307901f 2023-04-13 20:55:08,805 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1153 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/29dbcfeffc22405c832cf19b17950bed 2023-04-13 20:55:08,817 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1153 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/154c05d8c6424c319514ae8fc457d8a7 2023-04-13 20:55:08,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/25b0c4e062994564890b6f3a4307901f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/25b0c4e062994564890b6f3a4307901f 2023-04-13 20:55:08,823 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/25b0c4e062994564890b6f3a4307901f, entries=150, sequenceid=1153, filesize=11.9 K 2023-04-13 20:55:08,824 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/29dbcfeffc22405c832cf19b17950bed as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/29dbcfeffc22405c832cf19b17950bed 2023-04-13 20:55:08,827 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/29dbcfeffc22405c832cf19b17950bed, entries=150, sequenceid=1153, filesize=11.9 K 2023-04-13 20:55:08,828 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/154c05d8c6424c319514ae8fc457d8a7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/154c05d8c6424c319514ae8fc457d8a7 2023-04-13 20:55:08,831 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/154c05d8c6424c319514ae8fc457d8a7, entries=150, sequenceid=1153, filesize=11.9 K 2023-04-13 20:55:08,831 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5c92cecd6957c182fb4d0c5ce319c122 in 446ms, sequenceid=1153, compaction requested=true 2023-04-13 20:55:08,832 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:08,832 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:08,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:08,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,834 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:08,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d005952[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:08,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:08,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:08,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:08,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:08,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:08,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:08,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:08,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:08,842 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:08,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:08,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:08,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:08,842 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:08,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,844 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:08,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:08,845 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,846 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:08,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:08,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:08,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:08,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:08,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:08,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:08,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:08,849 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,849 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:08,849 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,849 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:08,849 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:08,949 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:08,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:08,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:08,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:08,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:08,954 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:08,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:08,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:08,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:08,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,956 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:08,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:08,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:08,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:08,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:08,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:08,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:08,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:08,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:08,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:08,957 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:08,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:08,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:08,958 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:08,958 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. started... 2023-04-13 20:55:08,958 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:08,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:08,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:08,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. as already flushing 2023-04-13 20:55:08,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419368992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419368995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:08,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:08,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419368996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,022 DEBUG [Thread-1809] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:54584 2023-04-13 20:55:09,022 DEBUG [Thread-1809] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:09,023 DEBUG [Thread-1813] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f8ff99f to 127.0.0.1:54584 2023-04-13 20:55:09,023 DEBUG [Thread-1813] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:09,024 DEBUG [Thread-1807] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:54584 2023-04-13 20:55:09,024 DEBUG [Thread-1807] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:09,024 DEBUG [Thread-1805] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:54584 2023-04-13 20:55:09,024 DEBUG [Thread-1805] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:09,026 DEBUG [Thread-1811] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:54584 2023-04-13 20:55:09,026 DEBUG [Thread-1811] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419369097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419369099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419369099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419369299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419369300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419369301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,367 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1170 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/61a07de5d6f04b289e450c1cd8f227b7 2023-04-13 20:55:09,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 456 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419369601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419369601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:09,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419369602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:09,775 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1170 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/9cf6d40b517b4a60a74f80c02955d039 2023-04-13 20:55:10,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48256 deadline: 1681419370103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48328 deadline: 1681419370104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:10,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48324 deadline: 1681419370106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,184 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1170 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/ca7ba92e614e48d29e0c3fe65a9c0479 2023-04-13 20:55:10,187 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/61a07de5d6f04b289e450c1cd8f227b7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/61a07de5d6f04b289e450c1cd8f227b7 2023-04-13 20:55:10,190 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/61a07de5d6f04b289e450c1cd8f227b7, entries=150, sequenceid=1170, filesize=11.9 K 2023-04-13 20:55:10,191 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/9cf6d40b517b4a60a74f80c02955d039 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/9cf6d40b517b4a60a74f80c02955d039 2023-04-13 20:55:10,194 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/9cf6d40b517b4a60a74f80c02955d039, entries=150, sequenceid=1170, filesize=11.9 K 2023-04-13 20:55:10,195 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/ca7ba92e614e48d29e0c3fe65a9c0479 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ca7ba92e614e48d29e0c3fe65a9c0479 2023-04-13 20:55:10,198 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ca7ba92e614e48d29e0c3fe65a9c0479, entries=150, sequenceid=1170, filesize=11.9 K 2023-04-13 20:55:10,198 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5c92cecd6957c182fb4d0c5ce319c122 in 1240ms, sequenceid=1170, compaction requested=true 2023-04-13 20:55:10,198 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:10,198 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:10,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:10,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,200 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:10,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:10,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49be44f3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:10,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:10,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:10,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:10,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:10,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,204 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:10,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:10,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:10,207 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:10,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:10,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:10,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:10,207 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:10,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:10,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:10,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:10,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:10,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:10,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:10,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:10,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:10,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:10,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:10,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:10,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:10,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:10,212 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:10,212 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:10,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:10,212 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,212 DEBUG [Thread-1804] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:10,212 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:10,212 DEBUG [Thread-1804] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:10,212 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:10,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:10,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:10,312 DEBUG [Thread-1804] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:11,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:11,108 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:55:11,109 DEBUG [Thread-1794] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:54584 2023-04-13 20:55:11,109 DEBUG [Thread-1794] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:11,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:11,112 DEBUG [Thread-1800] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:54584 2023-04-13 20:55:11,112 DEBUG [Thread-1800] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,115 DEBUG [Thread-1802] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:54584 2023-04-13 20:55:11,115 DEBUG [Thread-1802] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,517 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/002512e3d61c415f8d16b458649a467b 2023-04-13 20:55:11,782 DEBUG [Thread-1796] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:54584 2023-04-13 20:55:11,782 DEBUG [Thread-1796] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,788 DEBUG [Thread-1798] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:54584 2023-04-13 20:55:11,788 DEBUG [Thread-1798] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 250 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 305 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 220 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 6119 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 5740 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 5853 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 6097 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 5730 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:55:11,789 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:55:11,789 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:54584 2023-04-13 20:55:11,789 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:11,790 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:55:11,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:55:11,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:11,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:11,793 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419311793"}]},"ts":"1681419311793"} 2023-04-13 20:55:11,794 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:55:11,796 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:55:11,797 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, UNASSIGN}] 2023-04-13 20:55:11,798 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, UNASSIGN 2023-04-13 20:55:11,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5c92cecd6957c182fb4d0c5ce319c122, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:11,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419311798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419311798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419311798"}]},"ts":"1681419311798"} 2023-04-13 20:55:11,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:11,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:11,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/5f0e1f30321945ac89b0c71d3791f4ae 2023-04-13 20:55:11,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:11,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5c92cecd6957c182fb4d0c5ce319c122, disabling compactions & flushes 2023-04-13 20:55:11,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:12,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:12,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=1186 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/daed859890a44b5684970b09a539e202 2023-04-13 20:55:12,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/002512e3d61c415f8d16b458649a467b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/002512e3d61c415f8d16b458649a467b 2023-04-13 20:55:12,343 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/002512e3d61c415f8d16b458649a467b, entries=150, sequenceid=1186, filesize=11.9 K 2023-04-13 20:55:12,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/5f0e1f30321945ac89b0c71d3791f4ae as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5f0e1f30321945ac89b0c71d3791f4ae 2023-04-13 20:55:12,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5f0e1f30321945ac89b0c71d3791f4ae, entries=150, sequenceid=1186, filesize=11.9 K 2023-04-13 20:55:12,347 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/daed859890a44b5684970b09a539e202 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/daed859890a44b5684970b09a539e202 2023-04-13 20:55:12,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/daed859890a44b5684970b09a539e202, entries=150, sequenceid=1186, filesize=11.9 K 2023-04-13 20:55:12,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=26.84 KB/27480 for 5c92cecd6957c182fb4d0c5ce319c122 in 1242ms, sequenceid=1186, compaction requested=true 2023-04-13 20:55:12,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:12,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:12,350 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. because compaction request was cancelled 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:12,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. after waiting 0 ms 2023-04-13 20:55:12,351 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. because compaction request was cancelled 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:12,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:12,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5c92cecd6957c182fb4d0c5ce319c122 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 20:55:12,351 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:12,351 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. because compaction request was cancelled 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=A 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=B 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5c92cecd6957c182fb4d0c5ce319c122, store=C 2023-04-13 20:55:12,351 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:12,758 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1193 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/808c646bbfbf48d1afc31d142e1e062e 2023-04-13 20:55:12,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:13,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1193 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/7b854a7d30684da997a22673720c90ef 2023-04-13 20:55:13,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1193 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/b8b10f2e4da84b5aae4a264fc7022a16 2023-04-13 20:55:13,578 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/A/808c646bbfbf48d1afc31d142e1e062e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/808c646bbfbf48d1afc31d142e1e062e 2023-04-13 20:55:13,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/808c646bbfbf48d1afc31d142e1e062e, entries=150, sequenceid=1193, filesize=11.9 K 2023-04-13 20:55:13,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/B/7b854a7d30684da997a22673720c90ef as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7b854a7d30684da997a22673720c90ef 2023-04-13 20:55:13,585 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7b854a7d30684da997a22673720c90ef, entries=150, sequenceid=1193, filesize=11.9 K 2023-04-13 20:55:13,585 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/.tmp/C/b8b10f2e4da84b5aae4a264fc7022a16 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/b8b10f2e4da84b5aae4a264fc7022a16 2023-04-13 20:55:13,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/b8b10f2e4da84b5aae4a264fc7022a16, entries=150, sequenceid=1193, filesize=11.9 K 2023-04-13 20:55:13,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5c92cecd6957c182fb4d0c5ce319c122 in 1237ms, sequenceid=1193, compaction requested=true 2023-04-13 20:55:13,609 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/6e2e97bb34a845e58ca69a5f34e6d5de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d0f33fab4745491ead6b10b3522db6ef, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/aeafe16ebd2549269acf908fc74bb0f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f905f7ceb2034a5c9898bc66d35573d8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/998a3afe8f014113bc6fc855a0cb353a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c75a1aa28003426e868dc8232e40f5b4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/18e38a07d1784f798fdb8d734ee862a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fc902f853c3347d7b8b62f1635898c9a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/cd4c34cc10bd44adaf23a9adba1bd3da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071] to archive 2023-04-13 20:55:13,610 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:13,611 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d212c95abeda46b2ba7bc5b775454a48 2023-04-13 20:55:13,612 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ffbbaf2e2dab46b2a5cf4bc5f094c79f 2023-04-13 20:55:13,613 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c62b596dc07c47319325399fff903228 2023-04-13 20:55:13,614 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/667c3742ab9d41cc8c85cb4095a7319e 2023-04-13 20:55:13,615 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/6e2e97bb34a845e58ca69a5f34e6d5de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/6e2e97bb34a845e58ca69a5f34e6d5de 2023-04-13 20:55:13,616 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/466a6db429314631b41331253a0bde88 2023-04-13 20:55:13,617 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/54855fba22454e96905665744260956a 2023-04-13 20:55:13,618 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d0f33fab4745491ead6b10b3522db6ef to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/d0f33fab4745491ead6b10b3522db6ef 2023-04-13 20:55:13,619 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1f8a6b7849b04fd59a4ea056dc8d3d6b 2023-04-13 20:55:13,620 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/1db8ee8bff9c4bec81e0031b95a98d56 2023-04-13 20:55:13,621 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/aeafe16ebd2549269acf908fc74bb0f7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/aeafe16ebd2549269acf908fc74bb0f7 2023-04-13 20:55:13,622 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/63c480b575eb41d48f5de665a2ee78f9 2023-04-13 20:55:13,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8f5017ed4a44a6fab971c679e2bf77d 2023-04-13 20:55:13,623 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f905f7ceb2034a5c9898bc66d35573d8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f905f7ceb2034a5c9898bc66d35573d8 2023-04-13 20:55:13,624 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c9a220c40e654730b09f663a9e928976 2023-04-13 20:55:13,625 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/db1c3ba832a9428cbc00e7fbe359d6ff 2023-04-13 20:55:13,626 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fb37ecac3ad54f7fbaf0e9d6fadf0452 2023-04-13 20:55:13,627 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/998a3afe8f014113bc6fc855a0cb353a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/998a3afe8f014113bc6fc855a0cb353a 2023-04-13 20:55:13,628 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/7c5c26567bfe4ecd8eb10837def0c516 2023-04-13 20:55:13,629 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/decd5d8fb608496a88dd2ecfa365d3fd 2023-04-13 20:55:13,630 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c75a1aa28003426e868dc8232e40f5b4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c75a1aa28003426e868dc8232e40f5b4 2023-04-13 20:55:13,631 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/0723058fbc934915ae0c5ed3ef804dcf 2023-04-13 20:55:13,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ce66c85b0fa14e7e8ae31a81bdeab708 2023-04-13 20:55:13,632 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/96e986830b4c413e8d8de758d196800b 2023-04-13 20:55:13,633 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/ff342b243d224797bc3ff6f5cbbed1bf 2023-04-13 20:55:13,634 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/18e38a07d1784f798fdb8d734ee862a2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/18e38a07d1784f798fdb8d734ee862a2 2023-04-13 20:55:13,635 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/e9a24288b4a94dfcaee4102e02624162 2023-04-13 20:55:13,636 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/c8cd308ff73a45c592b05c8e53055e51 2023-04-13 20:55:13,637 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fc902f853c3347d7b8b62f1635898c9a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/fc902f853c3347d7b8b62f1635898c9a 2023-04-13 20:55:13,638 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/448b3f43faf74eb0bdc35f55d17de522 2023-04-13 20:55:13,639 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/14a8f18b3671480f81bfb082f0588e08 2023-04-13 20:55:13,640 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/cd4c34cc10bd44adaf23a9adba1bd3da to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/cd4c34cc10bd44adaf23a9adba1bd3da 2023-04-13 20:55:13,641 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/b962fd52ecf64cc3adc2aa3e595585f0 2023-04-13 20:55:13,642 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/709edc5eac994632a4dd7254a8af72cb 2023-04-13 20:55:13,643 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5db009d99be342cfb62928f9d978f6f6 2023-04-13 20:55:13,644 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/f3ad6e0945af417392ad69a52f778071 2023-04-13 20:55:13,647 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/98878c0179464da5b06e69e4e7852e5b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cfd901160bf54577a3e7d4cbcf701dbf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/af4f1ad4d37149c18683c17ae1507359, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f9c24648a7244c8cad06f912593854f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c2b7084debaf4c11b975f4fff709c92c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d7fe0feb3e7746b9bc6f71211679b030, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a1acfdc7acef4397a8b9704446bd4249, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/35f23a6a53da48a2b933f479c0be9c2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f3e608ac325346bdb2e36962495f9d91, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6] to archive 2023-04-13 20:55:13,648 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:13,649 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/43fa791836ad484da811bd790c41cb10 2023-04-13 20:55:13,650 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/169b779119be4e29aa4ead7795fd3fc5 2023-04-13 20:55:13,651 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/056cfa3ca15a4bd780605f72804344f4 2023-04-13 20:55:13,652 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/98878c0179464da5b06e69e4e7852e5b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/98878c0179464da5b06e69e4e7852e5b 2023-04-13 20:55:13,653 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6e6b7677df464777b64ac173911db907 2023-04-13 20:55:13,654 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1d8149d3644146d0b4f38c5102f84b40 2023-04-13 20:55:13,655 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cfd901160bf54577a3e7d4cbcf701dbf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cfd901160bf54577a3e7d4cbcf701dbf 2023-04-13 20:55:13,656 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7d9b85a68f8e4c5fa2233739c396166e 2023-04-13 20:55:13,657 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/b5c3582d1fed40109efdffd77577a1fd 2023-04-13 20:55:13,658 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/af4f1ad4d37149c18683c17ae1507359 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/af4f1ad4d37149c18683c17ae1507359 2023-04-13 20:55:13,660 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/abab2c24ab454ee0a05781be24fbf1f1 2023-04-13 20:55:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cb12e0debca64314b2d0e67c4885b7c4 2023-04-13 20:55:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/873b2fc817ff4c9abd88947941edaf0a 2023-04-13 20:55:13,662 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f9c24648a7244c8cad06f912593854f7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f9c24648a7244c8cad06f912593854f7 2023-04-13 20:55:13,663 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/0564b39e02664830be6a4869d18ea7b3 2023-04-13 20:55:13,664 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cf5c26c9706b43c1a4cfa16e23c99426 2023-04-13 20:55:13,665 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c2b7084debaf4c11b975f4fff709c92c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c2b7084debaf4c11b975f4fff709c92c 2023-04-13 20:55:13,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cba518928c81432ba66363c08900f5f2 2023-04-13 20:55:13,666 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/4e011056587245e99d05863464163cdf 2023-04-13 20:55:13,667 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/fd17154ccfde42b7b7094adb5b63fbe2 2023-04-13 20:55:13,668 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d7fe0feb3e7746b9bc6f71211679b030 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d7fe0feb3e7746b9bc6f71211679b030 2023-04-13 20:55:13,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/86c266cf5ab6467abbdda986edf666de 2023-04-13 20:55:13,669 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/1f08714238c6412a858309b12e596f2a 2023-04-13 20:55:13,670 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ac7a91a285284a7cb1479619bb74588a 2023-04-13 20:55:13,671 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5e52c8a562d743a3af9fd28295e7cbf7 2023-04-13 20:55:13,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a1acfdc7acef4397a8b9704446bd4249 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a1acfdc7acef4397a8b9704446bd4249 2023-04-13 20:55:13,672 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/8abdcb18d2e74c09b5fbdc457c92cf46 2023-04-13 20:55:13,673 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/c1c0338dddac4366bea7528d2a849604 2023-04-13 20:55:13,674 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/35f23a6a53da48a2b933f479c0be9c2a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/35f23a6a53da48a2b933f479c0be9c2a 2023-04-13 20:55:13,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/a09c7143e72341f79d6e728307341682 2023-04-13 20:55:13,675 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/d11b9e9338b144688a68d34869895793 2023-04-13 20:55:13,676 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f3e608ac325346bdb2e36962495f9d91 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/f3e608ac325346bdb2e36962495f9d91 2023-04-13 20:55:13,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/ca175a46021446458f641b0352035c84 2023-04-13 20:55:13,677 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/6a338b9ea12845f39bc954741662748a 2023-04-13 20:55:13,678 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/45819100d5604f0d9eaca517c6f08c1c 2023-04-13 20:55:13,679 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/180fa290e15e4e7a89c5e808fdfd16b6 2023-04-13 20:55:13,682 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/90aca0560a7d4285af1adcb87b3a5f33, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e81895fb25164ebe897e389e41244c94, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/60abbd41caf0477baff17d40276d0f2d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/037cf5b41103455d8c0a8612d55e20f0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1288a8099b0046e9a6d27a81af110c18, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/54944c59cca048e4aaa5c0abae0dfe05, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ffa9703a64324af5817d61423d239c7b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/8ca4b0bd512942abb556a2925f467473, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538] to archive 2023-04-13 20:55:13,683 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:13,684 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/28a0d7587e1a4000a571e76b3a2569bb 2023-04-13 20:55:13,685 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1d7e3a2e2dcf46d8b1e1c0a68eeefb59 2023-04-13 20:55:13,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9cf7ba9e386442a4ad3e42672f5d7e23 2023-04-13 20:55:13,686 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/2f42104c0ec04d05b483b4bbef864d93 2023-04-13 20:55:13,687 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/90aca0560a7d4285af1adcb87b3a5f33 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/90aca0560a7d4285af1adcb87b3a5f33 2023-04-13 20:55:13,688 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e112e7c712594dd88302726204c6de67 2023-04-13 20:55:13,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/c54708ca4a354d6d9653aef5af4eb855 2023-04-13 20:55:13,689 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e81895fb25164ebe897e389e41244c94 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e81895fb25164ebe897e389e41244c94 2023-04-13 20:55:13,690 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e433ce481b0d4d2c841fc3e7551a3053 2023-04-13 20:55:13,691 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/99fadd37c0914d03b63c5af79536aa76 2023-04-13 20:55:13,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1814485227084bdeb98e8b52423b61ab 2023-04-13 20:55:13,692 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/9c1a7802789544a594e95a7e32278c6d 2023-04-13 20:55:13,693 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/60abbd41caf0477baff17d40276d0f2d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/60abbd41caf0477baff17d40276d0f2d 2023-04-13 20:55:13,694 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a9dffaea55fd4c67bdad786a6df1a1d1 2023-04-13 20:55:13,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/338ad10df3c64568b5922bc60284f1fb 2023-04-13 20:55:13,695 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/037cf5b41103455d8c0a8612d55e20f0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/037cf5b41103455d8c0a8612d55e20f0 2023-04-13 20:55:13,696 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/3ec36c3185a84cfbaac4b762e342aec5 2023-04-13 20:55:13,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/89477fc1c78f4413b7f3abd904a25a08 2023-04-13 20:55:13,697 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/434cf000bca74def936e140e76448a8d 2023-04-13 20:55:13,698 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1288a8099b0046e9a6d27a81af110c18 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/1288a8099b0046e9a6d27a81af110c18 2023-04-13 20:55:13,699 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/509a0f17e1324e7b93351f06ff4161bc 2023-04-13 20:55:13,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/dd7c572d8c364a71b4f82aded1da7b90 2023-04-13 20:55:13,700 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/033ed26013be48a1b07e5d3b71de80f7 2023-04-13 20:55:13,701 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/830d2f5e3605407fa7af976df7834001 2023-04-13 20:55:13,702 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/54944c59cca048e4aaa5c0abae0dfe05 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/54944c59cca048e4aaa5c0abae0dfe05 2023-04-13 20:55:13,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e94397f8a6b944b8b845128e875a85d8 2023-04-13 20:55:13,703 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/929afd1649e14faaa91e39457d868213 2023-04-13 20:55:13,704 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ffa9703a64324af5817d61423d239c7b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ffa9703a64324af5817d61423d239c7b 2023-04-13 20:55:13,705 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/e097a64e996043debd66c524252b500a 2023-04-13 20:55:13,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5c971ec7d2dd4d459132f9968e95296b 2023-04-13 20:55:13,706 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/8ca4b0bd512942abb556a2925f467473 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/8ca4b0bd512942abb556a2925f467473 2023-04-13 20:55:13,707 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/a3586f5dfc324e9c8751394751c457bc 2023-04-13 20:55:13,708 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d82ebf71361492cadc57386381b5af3 2023-04-13 20:55:13,709 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/fbea4d957e51491292c096e599f5d379 2023-04-13 20:55:13,710 DEBUG [StoreCloser-TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/5d45789685f64dde9dd83be63c185538 2023-04-13 20:55:13,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/recovered.edits/1196.seqid, newMaxSeqId=1196, maxSeqId=1 2023-04-13 20:55:13,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122. 2023-04-13 20:55:13,714 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5c92cecd6957c182fb4d0c5ce319c122: 2023-04-13 20:55:13,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:13,716 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=5c92cecd6957c182fb4d0c5ce319c122, regionState=CLOSED 2023-04-13 20:55:13,716 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419313716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419313716"}]},"ts":"1681419313716"} 2023-04-13 20:55:13,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-13 20:55:13,718 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 5c92cecd6957c182fb4d0c5ce319c122, server=jenkins-hbase4.apache.org,36431,1681419217857 in 1.9180 sec 2023-04-13 20:55:13,719 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-13 20:55:13,719 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5c92cecd6957c182fb4d0c5ce319c122, UNASSIGN in 1.9210 sec 2023-04-13 20:55:13,719 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419313719"}]},"ts":"1681419313719"} 2023-04-13 20:55:13,720 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:55:13,722 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:55:13,723 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9320 sec 2023-04-13 20:55:13,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-13 20:55:13,897 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-13 20:55:13,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:55:13,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,899 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 20:55:13,900 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,903 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:13,905 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/recovered.edits] 2023-04-13 20:55:13,907 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/002512e3d61c415f8d16b458649a467b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/002512e3d61c415f8d16b458649a467b 2023-04-13 20:55:13,909 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/25b0c4e062994564890b6f3a4307901f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/25b0c4e062994564890b6f3a4307901f 2023-04-13 20:55:13,910 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/2607d5569c7541c08116373916df6d01 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/2607d5569c7541c08116373916df6d01 2023-04-13 20:55:13,911 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5d78d17845d64ce1a5020cc5e375ff61 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/5d78d17845d64ce1a5020cc5e375ff61 2023-04-13 20:55:13,911 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/61a07de5d6f04b289e450c1cd8f227b7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/61a07de5d6f04b289e450c1cd8f227b7 2023-04-13 20:55:13,912 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/808c646bbfbf48d1afc31d142e1e062e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/A/808c646bbfbf48d1afc31d142e1e062e 2023-04-13 20:55:13,914 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/29dbcfeffc22405c832cf19b17950bed to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/29dbcfeffc22405c832cf19b17950bed 2023-04-13 20:55:13,915 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/502e9584ffff469396b2891959fc3402 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/502e9584ffff469396b2891959fc3402 2023-04-13 20:55:13,915 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5f0e1f30321945ac89b0c71d3791f4ae to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/5f0e1f30321945ac89b0c71d3791f4ae 2023-04-13 20:55:13,916 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7b854a7d30684da997a22673720c90ef to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/7b854a7d30684da997a22673720c90ef 2023-04-13 20:55:13,917 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/9cf6d40b517b4a60a74f80c02955d039 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/9cf6d40b517b4a60a74f80c02955d039 2023-04-13 20:55:13,918 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cd6c66fc584648339ce4d6e23c659459 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/B/cd6c66fc584648339ce4d6e23c659459 2023-04-13 20:55:13,919 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/154c05d8c6424c319514ae8fc457d8a7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/154c05d8c6424c319514ae8fc457d8a7 2023-04-13 20:55:13,920 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/74252e34889d4f20970988b7154da488 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/74252e34889d4f20970988b7154da488 2023-04-13 20:55:13,921 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/b8b10f2e4da84b5aae4a264fc7022a16 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/b8b10f2e4da84b5aae4a264fc7022a16 2023-04-13 20:55:13,923 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ca7ba92e614e48d29e0c3fe65a9c0479 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/ca7ba92e614e48d29e0c3fe65a9c0479 2023-04-13 20:55:13,924 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/da85e5dbb9b1484f8b3e5d8a4cd65fca to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/da85e5dbb9b1484f8b3e5d8a4cd65fca 2023-04-13 20:55:13,925 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/daed859890a44b5684970b09a539e202 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/C/daed859890a44b5684970b09a539e202 2023-04-13 20:55:13,928 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/recovered.edits/1196.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122/recovered.edits/1196.seqid 2023-04-13 20:55:13,928 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/5c92cecd6957c182fb4d0c5ce319c122 2023-04-13 20:55:13,928 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:55:13,930 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,931 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:55:13,933 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:55:13,934 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,934 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:55:13,934 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419313934"}]},"ts":"9223372036854775807"} 2023-04-13 20:55:13,936 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:55:13,936 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5c92cecd6957c182fb4d0c5ce319c122, NAME => 'TestAcidGuarantees,,1681419286859.5c92cecd6957c182fb4d0c5ce319c122.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:55:13,936 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:55:13,936 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419313936"}]},"ts":"9223372036854775807"} 2023-04-13 20:55:13,937 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:55:13,939 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:13,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 42 msec 2023-04-13 20:55:14,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-13 20:55:14,001 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-13 20:55:14,012 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 224) - Thread LEAK? -, OpenFileDescriptor=575 (was 522) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=336 (was 343), ProcessCount=170 (was 170), AvailableMemoryMB=582 (was 631) 2023-04-13 20:55:14,022 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=170, AvailableMemoryMB=582 2023-04-13 20:55:14,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:55:14,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:55:14,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:14,026 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:55:14,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-13 20:55:14,026 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:55:14,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:14,027 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:14,028 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 empty. 2023-04-13 20:55:14,028 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:14,028 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:55:14,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:14,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:14,435 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:55:14,436 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 53d52adae37ccaf3c7691f3ca2bf72d5, NAME => 'TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:55:14,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 53d52adae37ccaf3c7691f3ca2bf72d5, disabling compactions & flushes 2023-04-13 20:55:14,842 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. after waiting 0 ms 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:14,842 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:14,842 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:14,844 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:55:14,845 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419314845"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419314845"}]},"ts":"1681419314845"} 2023-04-13 20:55:14,846 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:55:14,847 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:55:14,847 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419314847"}]},"ts":"1681419314847"} 2023-04-13 20:55:14,847 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:55:14,853 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, ASSIGN}] 2023-04-13 20:55:14,854 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, ASSIGN 2023-04-13 20:55:14,855 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:55:15,006 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:15,006 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419315006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419315006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419315006"}]},"ts":"1681419315006"} 2023-04-13 20:55:15,007 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:15,162 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:15,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 53d52adae37ccaf3c7691f3ca2bf72d5, NAME => 'TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:55:15,162 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:55:15,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,164 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,165 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A 2023-04-13 20:55:15,165 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A 2023-04-13 20:55:15,165 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:15,165 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName A 2023-04-13 20:55:15,166 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:15,166 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,167 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B 2023-04-13 20:55:15,167 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B 2023-04-13 20:55:15,167 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:15,167 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName B 2023-04-13 20:55:15,168 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:15,168 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,169 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C 2023-04-13 20:55:15,169 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C 2023-04-13 20:55:15,169 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:15,169 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName C 2023-04-13 20:55:15,169 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:15,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:15,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:55:15,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:15,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:55:15,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 53d52adae37ccaf3c7691f3ca2bf72d5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71769016, jitterRate=0.06944167613983154}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:55:15,175 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:15,175 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., pid=37, masterSystemTime=1681419315159 2023-04-13 20:55:15,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:15,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:15,177 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:15,177 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419315177"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419315177"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419315177"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419315177"}]},"ts":"1681419315177"} 2023-04-13 20:55:15,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-13 20:55:15,179 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 in 171 msec 2023-04-13 20:55:15,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-13 20:55:15,180 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, ASSIGN in 326 msec 2023-04-13 20:55:15,180 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:55:15,180 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419315180"}]},"ts":"1681419315180"} 2023-04-13 20:55:15,181 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:55:15,183 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:55:15,184 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-04-13 20:55:16,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-13 20:55:16,131 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-13 20:55:16,131 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:16,136 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:16,137 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:16,139 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51138, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:16,140 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:55:16,141 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55914, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:55:16,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:55:16,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:55:16,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:16,552 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:55:16,553 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:55:16,553 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:55:16,555 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 20:55:16,557 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, REOPEN/MOVE}] 2023-04-13 20:55:16,558 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, REOPEN/MOVE 2023-04-13 20:55:16,558 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:16,558 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419316558"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419316558"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419316558"}]},"ts":"1681419316558"} 2023-04-13 20:55:16,559 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:16,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:16,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 53d52adae37ccaf3c7691f3ca2bf72d5, disabling compactions & flushes 2023-04-13 20:55:16,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:16,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:16,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. after waiting 0 ms 2023-04-13 20:55:16,712 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:16,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 20:55:16,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:16,715 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:16,715 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 53d52adae37ccaf3c7691f3ca2bf72d5 to self. 2023-04-13 20:55:16,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:16,717 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=CLOSED 2023-04-13 20:55:16,717 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419316717"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419316717"}]},"ts":"1681419316717"} 2023-04-13 20:55:16,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-13 20:55:16,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 in 159 msec 2023-04-13 20:55:16,719 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=true 2023-04-13 20:55:16,870 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:16,870 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419316869"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419316869"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419316869"}]},"ts":"1681419316869"} 2023-04-13 20:55:16,871 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:17,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:17,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 53d52adae37ccaf3c7691f3ca2bf72d5, NAME => 'TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:55:17,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:55:17,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,026 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,027 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A 2023-04-13 20:55:17,027 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A 2023-04-13 20:55:17,028 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:17,028 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName A 2023-04-13 20:55:17,029 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:17,029 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,030 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B 2023-04-13 20:55:17,030 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B 2023-04-13 20:55:17,030 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:17,030 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName B 2023-04-13 20:55:17,031 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:17,031 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,031 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C 2023-04-13 20:55:17,032 DEBUG [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C 2023-04-13 20:55:17,032 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:17,032 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 53d52adae37ccaf3c7691f3ca2bf72d5 columnFamilyName C 2023-04-13 20:55:17,032 INFO [StoreOpener-53d52adae37ccaf3c7691f3ca2bf72d5-1] regionserver.HStore(310): Store=53d52adae37ccaf3c7691f3ca2bf72d5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:17,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:17,033 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,035 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:55:17,037 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 53d52adae37ccaf3c7691f3ca2bf72d5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58894637, jitterRate=-0.12240152060985565}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:55:17,038 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:17,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., pid=42, masterSystemTime=1681419317022 2023-04-13 20:55:17,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:17,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:17,040 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,040 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419317040"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419317040"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419317040"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419317040"}]},"ts":"1681419317040"} 2023-04-13 20:55:17,042 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-13 20:55:17,043 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 in 170 msec 2023-04-13 20:55:17,044 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-13 20:55:17,044 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, REOPEN/MOVE in 486 msec 2023-04-13 20:55:17,046 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-13 20:55:17,046 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-13 20:55:17,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-13 20:55:17,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-13 20:55:17,050 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,055 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,056 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,061 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,061 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,069 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,069 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,072 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,072 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,076 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,076 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,081 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,081 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,085 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,085 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,089 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,090 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,093 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,093 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7eae4191 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:17,097 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ce79633, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:17,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:17,104 DEBUG [hconnection-0x6194d01b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,104 DEBUG [hconnection-0x63f649a1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,105 DEBUG [hconnection-0x56eab472-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,105 DEBUG [hconnection-0x3f43a067-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,105 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,105 DEBUG [hconnection-0x3d4527c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,108 DEBUG [hconnection-0x2681cf5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,109 DEBUG [hconnection-0x4bf25941-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,109 DEBUG [hconnection-0x2f28ba44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,110 DEBUG [hconnection-0x1e2d19af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,110 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,110 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,110 DEBUG [hconnection-0x787c076e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:17,113 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51172, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:17,117 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51184, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:17,118 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:17,118 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:17,118 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:17,118 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:17,118 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51164, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,118 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:17,118 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:17,118 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51194, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,118 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51228, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:17,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:17,120 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,120 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51212, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:17,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51202, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:17,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:17,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:17,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:17,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:17,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:17,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:17,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:17,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:17,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:17,127 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:17,127 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:17,127 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:17,129 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:17,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:17,133 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419317177, max timestamp=1681419317183Num uniques 859; 2023-04-13 20:55:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419317177, max timestamp=1681419317183Num uniques 859; 2023-04-13 20:55:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419317177, max timestamp=1681419317183Num uniques 859; 2023-04-13 20:55:17,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419377206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419377208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419377209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419377210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,211 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419377211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681419317201, max timestamp=1681419317210Num uniques 1068; 2023-04-13 20:55:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-13 20:55:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681419317201, max timestamp=1681419317210Num uniques 1068; 2023-04-13 20:55:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1681419317201, max timestamp=1681419317210Num uniques 1068; 2023-04-13 20:55:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-13 20:55:17,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=16848, suffix heap size=139040,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=368 2023-04-13 20:55:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419377365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419377365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419377366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419377366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419377367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419377476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419377476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419377476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419377477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419377477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,552 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413191435be04da4a229aa16ef6a5c80ee2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413191435be04da4a229aa16ef6a5c80ee2 2023-04-13 20:55:17,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419377682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419377682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419377683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419377683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419377684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,958 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/c3241d1fc5f143cfa29f2dc4178d2dbb 2023-04-13 20:55:17,968 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/61834871f3724d8fb31909418c1bb5d6 2023-04-13 20:55:17,990 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419377986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419377986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419377987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419377987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:17,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:17,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419377990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,377 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b1a0aafcce3347f1921ac6cd6f0afa9b 2023-04-13 20:55:18,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/c3241d1fc5f143cfa29f2dc4178d2dbb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb 2023-04-13 20:55:18,385 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb, entries=150, sequenceid=12, filesize=25.1 K 2023-04-13 20:55:18,386 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/61834871f3724d8fb31909418c1bb5d6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6 2023-04-13 20:55:18,392 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6, entries=150, sequenceid=12, filesize=11.6 K 2023-04-13 20:55:18,393 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b1a0aafcce3347f1921ac6cd6f0afa9b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b 2023-04-13 20:55:18,404 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b, entries=150, sequenceid=12, filesize=11.6 K 2023-04-13 20:55:18,405 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1277ms, sequenceid=12, compaction requested=false 2023-04-13 20:55:18,405 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:18,405 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,405 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:18,406 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:18,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:18,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,410 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,410 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:18,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7506cd7c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:18,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:18,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:18,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,418 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:18,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:18,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:18,423 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:18,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:18,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:18,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:18,423 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:18,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,424 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,425 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:18,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:18,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,431 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:18,431 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:18,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:18,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419317341, max timestamp=1681419318496Num uniques 1227; 2023-04-13 20:55:18,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-13 20:55:18,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419317341, max timestamp=1681419318496Num uniques 1227; 2023-04-13 20:55:18,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-13 20:55:18,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419317341, max timestamp=1681419318496Num uniques 1227; 2023-04-13 20:55:18,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=56192, new segment data size=8102, suffix heap size=155576,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1227, new segment cells count=177 2023-04-13 20:55:18,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=40.25 KB heapSize=104.39 KB 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:18,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:18,532 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:18,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:18,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:18,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:18,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:18,541 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:18,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:18,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:18,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681419318523, max timestamp=1681419318540Num uniques 859; 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419318523, max timestamp=1681419318540Num uniques 859; 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419318523, max timestamp=1681419318540Num uniques 859; 2023-04-13 20:55:18,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:18,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,543 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:18,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:18,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:18,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:18,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:18,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:18,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:18,550 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:18,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:18,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76560d64[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:18,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:18,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:18,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:18,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:18,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:18,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:18,563 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:18,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:18,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:18,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:18,563 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:18,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,571 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:18,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:18,571 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,572 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:18,572 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:18,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419378566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419378568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419378570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419378574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419378575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,672 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:18,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:18,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:18,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:18,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:18,676 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:18,676 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:18,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:18,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,679 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,680 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:18,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:18,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:18,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419378676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:18,681 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,681 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:18,681 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:18,681 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:18,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:18,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419378679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419378680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:18,685 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:18,685 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c31035[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:18,686 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:18,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:18,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:18,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:18,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419378684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,690 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:18,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:18,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:18,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:18,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:18,691 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:18,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419378686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,696 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,696 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,696 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:18,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,702 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:18,702 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:18,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,702 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:18,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,802 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:18,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:18,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:18,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:18,808 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:18,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:18,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:18,808 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,809 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,810 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:18,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:18,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:18,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:18,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:18,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:18,812 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:18,812 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:18,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:18,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:18,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:18,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12614022[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:18,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:18,818 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:18,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:18,820 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:18,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:18,824 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:18,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:18,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:18,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:18,824 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:18,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:18,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,832 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:18,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,832 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:18,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419378882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419378884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419378884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419378892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:18,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419378892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,929 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413b0495da185b0406babb6deeb87b8641f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b0495da185b0406babb6deeb87b8641f 2023-04-13 20:55:18,936 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:18,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:18,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:18,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:18,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:18,940 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:18,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:18,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:18,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,941 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,941 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:18,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:18,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:18,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:18,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:18,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:18,943 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:18,944 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:18,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:18,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51cf123f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:18,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,949 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:18,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,952 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:18,957 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:18,957 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:18,957 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:18,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:18,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:18,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,961 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:18,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:18,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:18,964 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,964 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:18,964 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:18,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:18,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:18,965 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:18,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:18,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,065 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,069 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,071 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,072 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,073 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b4084f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,085 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,086 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,089 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,090 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,090 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,093 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:19,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,099 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,099 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,192 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419379189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419379190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419379190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,199 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,203 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419379198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,204 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419379200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,206 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,206 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,207 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,207 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,207 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2647f034[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,212 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,214 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,217 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,217 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,219 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,220 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,223 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,224 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,224 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,243 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:55:19,324 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,327 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,328 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,328 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,330 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,331 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,331 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,331 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,331 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,331 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,334 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ebfd48e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,336 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,338 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,338 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,340 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,340 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,340 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,340 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,340 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,341 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=70, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/4d408a4c60834717aa521a5f2505ea23 2023-04-13 20:55:19,341 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,342 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:19,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,355 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,356 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,356 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,356 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,456 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,459 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,464 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c8678c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,467 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,467 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,468 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,471 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,471 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,471 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,471 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,471 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,474 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,474 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,474 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,475 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,480 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:19,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,480 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,480 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,581 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,585 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2089e34a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,597 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,600 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,600 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,600 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,600 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,600 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,608 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,616 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,616 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,616 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,615 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:19,616 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,616 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,616 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419379693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419379695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419379696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419379705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:19,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419379709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,716 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,720 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,728 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,729 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,729 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,729 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:19,729 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:19,729 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:19,731 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:19,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d6613f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:19,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:19,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,735 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,735 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:19,735 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,735 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:19,735 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:19,735 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:19,737 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,737 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:19,741 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:19,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:19,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:19,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:19,741 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:19,743 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,743 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:19,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,743 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:19,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:19,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,746 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:19,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:19,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,747 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:19,747 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:19,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/9fe857ffc4ab4e5b976576c95422d707 2023-04-13 20:55:19,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/0e72455e97a240bfaf5ff453665b665e 2023-04-13 20:55:19,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/4d408a4c60834717aa521a5f2505ea23 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23 2023-04-13 20:55:19,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23, entries=350, sequenceid=70, filesize=52.3 K 2023-04-13 20:55:19,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/9fe857ffc4ab4e5b976576c95422d707 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707 2023-04-13 20:55:19,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707, entries=150, sequenceid=70, filesize=11.6 K 2023-04-13 20:55:19,780 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/0e72455e97a240bfaf5ff453665b665e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e 2023-04-13 20:55:19,784 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e, entries=150, sequenceid=70, filesize=11.6 K 2023-04-13 20:55:19,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~124.27 KB/127248, currentSize=140.89 KB/144270 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1284ms, sequenceid=70, compaction requested=false 2023-04-13 20:55:19,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:19,847 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:19,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:19,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:19,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:19,850 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:19,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:19,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:19,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,857 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:19,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:19,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:19,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:19,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:19,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:19,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:19,858 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:19,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:19,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:19,859 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:19,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:20,275 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041374efd44f00314a72a16699790e99d783 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041374efd44f00314a72a16699790e99d783 2023-04-13 20:55:20,286 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=107, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/068abe1c8df642a985045e3d4a1fa1cb 2023-04-13 20:55:20,698 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/de6647c591e34d3d841590523a7400a5 2023-04-13 20:55:20,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:20,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:20,716 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/6ff7b76beb7b46b7a84485b3609f7f4c 2023-04-13 20:55:20,721 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/068abe1c8df642a985045e3d4a1fa1cb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb 2023-04-13 20:55:20,725 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb, entries=150, sequenceid=107, filesize=25.1 K 2023-04-13 20:55:20,726 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/de6647c591e34d3d841590523a7400a5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5 2023-04-13 20:55:20,729 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5, entries=150, sequenceid=107, filesize=11.6 K 2023-04-13 20:55:20,730 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/6ff7b76beb7b46b7a84485b3609f7f4c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c 2023-04-13 20:55:20,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419380722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419380723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419380724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,735 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c, entries=150, sequenceid=107, filesize=11.6 K 2023-04-13 20:55:20,735 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 876ms, sequenceid=107, compaction requested=true 2023-04-13 20:55:20,735 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:20,735 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:20,735 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:20,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:20,737 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,737 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,738 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:20,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790dd92f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:20,739 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:20,739 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,740 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:20,741 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:20,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:20,743 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:20,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:20,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:20,743 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:20,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:20,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:20,746 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:20,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:20,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:20,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:20,747 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:20,748 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,748 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,748 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:20,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,749 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,751 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:20,754 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:20,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:20,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:20,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:20,754 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:20,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,754 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,754 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,754 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:20,754 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:20,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:20,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419320811, max timestamp=1681419320829Num uniques 859; 2023-04-13 20:55:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419320811, max timestamp=1681419320829Num uniques 859; 2023-04-13 20:55:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419320811, max timestamp=1681419320829Num uniques 859; 2023-04-13 20:55:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:20,854 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:20,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:20,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:20,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:20,860 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:20,860 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:20,860 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:20,860 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:20,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,862 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:20,862 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:20,862 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,862 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:20,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:20,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:20,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:20,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:20,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:20,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:20,870 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:20,870 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,875 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,876 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:20,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dad3585[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:20,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:20,878 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:20,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:20,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:20,879 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:20,880 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:20,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,881 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:20,883 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:20,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:20,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:20,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:20,884 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:20,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419380873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419380876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419380877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:20,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419380877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:20,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:20,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:20,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:20,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:20,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:20,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:20,891 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:20,891 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:20,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:20,992 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:20,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:20,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419380986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419380986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419380988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:20,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419380988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:20,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:20,995 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:20,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:20,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:20,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:20,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,995 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,996 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:20,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:20,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:20,997 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,997 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:20,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:20,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:20,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:20,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:20,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:20,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:20,998 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:20,999 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d100c9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,003 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,005 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,007 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,007 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,012 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:21,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,013 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,013 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,113 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,117 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,120 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,120 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,132 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,134 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,134 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48844e35[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,137 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,138 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,138 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,140 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,142 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,142 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,145 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,145 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,145 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,150 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,150 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,177 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041385e0d3bff7244f75afc6767f1ddeccdc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385e0d3bff7244f75afc6767f1ddeccdc 2023-04-13 20:55:21,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419381193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,206 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419381194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419381194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419381194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,250 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,253 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,253 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,254 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,255 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,256 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,256 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,257 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,257 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,257 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,257 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,258 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,258 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,259 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,260 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fd732ab[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,260 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,262 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,262 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,266 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,268 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,269 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,274 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,274 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,374 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,378 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,379 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,379 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,383 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1225c9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,384 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,384 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,386 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,389 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,391 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,391 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,391 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,391 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,391 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,393 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,397 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,397 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,397 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,398 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,398 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,498 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,501 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,504 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,504 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419381507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419381508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@510fa92a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,510 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,510 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,512 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,515 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,515 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,515 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,515 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,515 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419381508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:21,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419381509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,518 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,519 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,522 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,522 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,583 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/bceb93650cec499c84a147a2483dbc68 2023-04-13 20:55:21,622 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,625 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,626 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,627 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,628 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b10b0d8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,643 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,644 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,645 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,647 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,647 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,647 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,651 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,651 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,651 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,651 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,653 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,655 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,655 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,655 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,755 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,759 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,760 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,761 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,763 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,764 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,764 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db48f7b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,768 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,768 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,768 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,769 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,773 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,773 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,773 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,776 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,777 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,780 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:21,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,780 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,781 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,781 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,881 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:21,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:21,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:21,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:21,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:21,884 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:21,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:21,884 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:21,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,885 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,887 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:21,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:21,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:21,897 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,897 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:21,897 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:21,897 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:21,897 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:21,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,899 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:21,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5782d569[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:21,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:21,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:21,903 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:21,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:21,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,905 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:21,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:21,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:21,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:21,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:21,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:21,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:21,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:21,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:21,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:21,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:21,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:21,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:21,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:21,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:21,914 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:21,914 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:21,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:21,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:21,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/31553bd408334ac0880e5bffad2a3979 2023-04-13 20:55:22,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/81eaec7f24014cf2b831e00fa186f216 2023-04-13 20:55:22,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/bceb93650cec499c84a147a2483dbc68 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68 2023-04-13 20:55:22,014 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68, entries=350, sequenceid=118, filesize=52.3 K 2023-04-13 20:55:22,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419382010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/31553bd408334ac0880e5bffad2a3979 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979 2023-04-13 20:55:22,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,017 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,019 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979, entries=150, sequenceid=118, filesize=11.6 K 2023-04-13 20:55:22,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,020 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/81eaec7f24014cf2b831e00fa186f216 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216 2023-04-13 20:55:22,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,021 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,021 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,021 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,021 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419382013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216, entries=150, sequenceid=118, filesize=11.6 K 2023-04-13 20:55:22,023 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,023 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1280ms, sequenceid=118, compaction requested=true 2023-04-13 20:55:22,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:22,024 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,024 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:22,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:22,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,025 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 158512 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,025 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:22,025 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,025 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:22,025 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=154.8 K 2023-04-13 20:55:22,025 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:22,025 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,026 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=46.4 K 2023-04-13 20:55:22,026 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c3241d1fc5f143cfa29f2dc4178d2dbb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419317127 2023-04-13 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,026 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 61834871f3724d8fb31909418c1bb5d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419317127 2023-04-13 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7be8604a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,026 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 4d408a4c60834717aa521a5f2505ea23, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681419317341 2023-04-13 20:55:22,026 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9fe857ffc4ab4e5b976576c95422d707, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681419317341 2023-04-13 20:55:22,027 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 068abe1c8df642a985045e3d4a1fa1cb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681419318564 2023-04-13 20:55:22,027 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting de6647c591e34d3d841590523a7400a5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681419318564 2023-04-13 20:55:22,027 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting bceb93650cec499c84a147a2483dbc68, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:22,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:22,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:22,028 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:22,028 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 31553bd408334ac0880e5bffad2a3979, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:22,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,031 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,031 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,034 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,034 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,035 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,040 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:22,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,045 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#359 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,054 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,054 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,054 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,057 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/e45bf15223de44d18caa0f9965a5b59d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e45bf15223de44d18caa0f9965a5b59d 2023-04-13 20:55:22,061 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into e45bf15223de44d18caa0f9965a5b59d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:22,061 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:22,061 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=12, startTime=1681419322024; duration=0sec 2023-04-13 20:55:22,061 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:22,061 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:22,061 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413da80283b715c4c00935c4890c626c517 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413da80283b715c4c00935c4890c626c517 2023-04-13 20:55:22,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419382052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,069 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:22,069 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:22,069 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,070 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=46.4 K 2023-04-13 20:55:22,071 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b1a0aafcce3347f1921ac6cd6f0afa9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419317127 2023-04-13 20:55:22,071 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0e72455e97a240bfaf5ff453665b665e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1681419317341 2023-04-13 20:55:22,071 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6ff7b76beb7b46b7a84485b3609f7f4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1681419318564 2023-04-13 20:55:22,072 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 81eaec7f24014cf2b831e00fa186f216, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:22,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419382064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,088 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#361 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:22,090 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=155, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/5482cddb894248e8ab726478ea0a54d4 2023-04-13 20:55:22,154 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,159 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,161 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419382165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419382175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4540899f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,187 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,195 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,201 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,201 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,209 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,209 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:22,309 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,315 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,317 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,320 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,320 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,320 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,320 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,320 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,320 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,320 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,320 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,320 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,321 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,324 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c76191b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,328 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,332 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,332 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,337 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,338 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,339 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,339 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,339 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,340 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,345 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,345 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,346 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,346 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,346 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:22,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419382373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419382383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,446 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,450 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,454 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,454 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,454 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,454 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,454 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,455 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1eba61[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,469 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,470 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,479 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,479 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,479 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:22,493 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/71cb3d037b7a439381fa023283f30d02 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/71cb3d037b7a439381fa023283f30d02 2023-04-13 20:55:22,498 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into 71cb3d037b7a439381fa023283f30d02(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:22,498 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:22,498 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=12, startTime=1681419322024; duration=0sec 2023-04-13 20:55:22,498 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:22,538 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/9ca124a56fd44a89a21f3aa590b812da as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9ca124a56fd44a89a21f3aa590b812da 2023-04-13 20:55:22,539 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/bd4a79904948438f90285b8cbe41bf11 2023-04-13 20:55:22,546 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into 9ca124a56fd44a89a21f3aa590b812da(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:22,546 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:22,546 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=12, startTime=1681419322024; duration=0sec 2023-04-13 20:55:22,546 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:22,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/ef52b306df4545eca526584fc9be8473 2023-04-13 20:55:22,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/5482cddb894248e8ab726478ea0a54d4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4 2023-04-13 20:55:22,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4, entries=200, sequenceid=155, filesize=32.1 K 2023-04-13 20:55:22,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/bd4a79904948438f90285b8cbe41bf11 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11 2023-04-13 20:55:22,580 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,583 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11, entries=150, sequenceid=155, filesize=11.8 K 2023-04-13 20:55:22,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/ef52b306df4545eca526584fc9be8473 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473 2023-04-13 20:55:22,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473, entries=150, sequenceid=155, filesize=11.8 K 2023-04-13 20:55:22,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 564ms, sequenceid=155, compaction requested=false 2023-04-13 20:55:22,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc7fb8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,593 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,599 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,599 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:22,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,608 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,608 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:22,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:22,686 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:22,699 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304138e69dd1987654c089ccc490747bfc8b5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e69dd1987654c089ccc490747bfc8b5 2023-04-13 20:55:22,708 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,710 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,712 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,715 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,716 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4811aef[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,719 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,719 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,723 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,723 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,726 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,729 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,729 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419322724, max timestamp=1681419322742Num uniques 859; 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419322724, max timestamp=1681419322742Num uniques 859; 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419322724, max timestamp=1681419322742Num uniques 859; 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:22,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:22,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419382776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419382777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419382788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,830 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,832 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:22,838 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,839 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83375a5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:22,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:22,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:22,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:22,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:22,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:22,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:22,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:22,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:22,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:22,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:22,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:22,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,860 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,864 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:22,865 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,867 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,867 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:22,868 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:22,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:22,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:22,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,868 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:22,868 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:22,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:22,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:22,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419382889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419382889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:22,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419382897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,968 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:22,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:22,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:22,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:22,976 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:22,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:22,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:22,986 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,986 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,988 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:22,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:22,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:22,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:22,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:22,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:22,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:22,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:22,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:22,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:22,996 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:22,996 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,001 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:23,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f61b70[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:23,005 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:23,005 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,006 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:23,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,007 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:23,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:23,013 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:23,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:23,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:23,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:23,013 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:23,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,017 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419383017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:23,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:23,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,022 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:23,022 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:23,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419383031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419383098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419383098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,105 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=169, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/054afbed8b014a799377ac19366244ea 2023-04-13 20:55:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419383103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,122 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:23,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:23,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:23,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:23,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:23,126 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,126 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,127 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:23,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:23,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:23,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:23,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:23,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:23,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,132 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:23,136 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,136 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:23,136 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:23,136 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,138 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:23,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:23,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26618569[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:23,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:23,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,141 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:23,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,141 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:23,141 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:23,144 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:23,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:23,147 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:23,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:23,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:23,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:23,147 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:23,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:23,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:23,161 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:23,162 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:23,262 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:23,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:23,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:23,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:23,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:23,268 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:23,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:23,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:23,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:23,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2abbb528[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:23,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:23,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:23,285 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,285 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:23,288 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:23,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:23,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:23,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:23,288 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:23,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:23,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:23,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,294 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:23,294 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:23,394 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:23,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:23,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:23,397 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:23,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:23,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:23,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,399 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,399 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,399 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:23,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:23,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:23,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:23,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:23,403 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:23,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39366387[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:23,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:23,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419383405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419383406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:23,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,410 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:23,413 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:23,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:23,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:23,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:23,413 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:23,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419383408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,415 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,415 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,415 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,419 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,419 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:23,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:23,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,420 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:23,420 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,520 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:23,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:23,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/4654b16aa4014020bff4f570260cc3c6 2023-04-13 20:55:23,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:23,523 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:23,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:23,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:23,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,524 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,526 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,526 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:23,527 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:23,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:23,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:23,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:23,528 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:23,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:23,529 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/71e79e9671994b87aaca87f0b5d91292 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:23,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92bc24a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:23,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:23,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:23,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:23,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/054afbed8b014a799377ac19366244ea as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea 2023-04-13 20:55:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:23,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:23,537 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:23,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:23,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:23,537 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:23,537 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:23,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea, entries=200, sequenceid=169, filesize=32.1 K 2023-04-13 20:55:23,539 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/4654b16aa4014020bff4f570260cc3c6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6 2023-04-13 20:55:23,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:23,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,540 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,541 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:23,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,541 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6, entries=150, sequenceid=169, filesize=11.8 K 2023-04-13 20:55:23,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:23,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:23,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:23,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:23,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:23,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/71e79e9671994b87aaca87f0b5d91292 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292 2023-04-13 20:55:23,545 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:23,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,545 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:23,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:23,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:23,545 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292, entries=150, sequenceid=169, filesize=11.8 K 2023-04-13 20:55:23,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 863ms, sequenceid=169, compaction requested=true 2023-04-13 20:55:23,549 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:23,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:23,549 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:23,549 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:23,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:23,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:23,550 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91656 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:23,550 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:23,550 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:23,550 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:23,550 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,550 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,550 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/71cb3d037b7a439381fa023283f30d02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=89.5 K 2023-04-13 20:55:23,550 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e45bf15223de44d18caa0f9965a5b59d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.3 K 2023-04-13 20:55:23,551 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 71cb3d037b7a439381fa023283f30d02, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:23,551 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e45bf15223de44d18caa0f9965a5b59d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:23,551 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5482cddb894248e8ab726478ea0a54d4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681419320863 2023-04-13 20:55:23,551 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bd4a79904948438f90285b8cbe41bf11, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681419320863 2023-04-13 20:55:23,551 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 054afbed8b014a799377ac19366244ea, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322049 2023-04-13 20:55:23,553 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4654b16aa4014020bff4f570260cc3c6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322050 2023-04-13 20:55:23,559 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:23,561 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#367 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:23,572 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/dcef5e7896804e72a857d5bbf4f4629e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dcef5e7896804e72a857d5bbf4f4629e 2023-04-13 20:55:23,576 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into dcef5e7896804e72a857d5bbf4f4629e(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:23,576 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:23,576 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419323549; duration=0sec 2023-04-13 20:55:23,576 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:23,576 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:23,578 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36096 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:23,578 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:23,578 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,578 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9ca124a56fd44a89a21f3aa590b812da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.3 K 2023-04-13 20:55:23,578 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9ca124a56fd44a89a21f3aa590b812da, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1681419320710 2023-04-13 20:55:23,579 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ef52b306df4545eca526584fc9be8473, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1681419320863 2023-04-13 20:55:23,579 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 71e79e9671994b87aaca87f0b5d91292, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322050 2023-04-13 20:55:23,582 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#369 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:23,589 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/0e2227fc57f74697973dd4072ac621b3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e2227fc57f74697973dd4072ac621b3 2023-04-13 20:55:23,593 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into 0e2227fc57f74697973dd4072ac621b3(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:23,593 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:23,593 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419323549; duration=0sec 2023-04-13 20:55:23,593 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:23,645 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:23,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:23,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:23,648 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:23,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:23,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:23,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,648 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,649 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:23,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:23,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:23,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:23,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:23,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:23,651 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:23,651 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:23,651 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:23,652 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:23,652 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:23,675 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304132337d3c333134048b15f28e478ce60ea to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132337d3c333134048b15f28e478ce60ea 2023-04-13 20:55:23,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:23,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:23,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419383930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419383930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:23,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419383939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:23,982 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/b7a8d844cb464042a8c73d473eadecfd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b7a8d844cb464042a8c73d473eadecfd 2023-04-13 20:55:23,987 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into b7a8d844cb464042a8c73d473eadecfd(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:23,987 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:23,987 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419323549; duration=0sec 2023-04-13 20:55:23,987 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:24,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419384041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419384041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419384050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,082 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/fa309cbc1e0a453e819391a245f6755d 2023-04-13 20:55:24,098 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/2a71bd48d6fd4323bca35f27a6a06cee 2023-04-13 20:55:24,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419384243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419384243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,260 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419384257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,530 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/cf04e3775d87467598bcad76a5859593 2023-04-13 20:55:24,533 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/fa309cbc1e0a453e819391a245f6755d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d 2023-04-13 20:55:24,536 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d, entries=150, sequenceid=206, filesize=25.3 K 2023-04-13 20:55:24,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/2a71bd48d6fd4323bca35f27a6a06cee as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee 2023-04-13 20:55:24,548 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee, entries=150, sequenceid=206, filesize=11.8 K 2023-04-13 20:55:24,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/cf04e3775d87467598bcad76a5859593 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593 2023-04-13 20:55:24,553 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593, entries=150, sequenceid=206, filesize=11.8 K 2023-04-13 20:55:24,554 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 902ms, sequenceid=206, compaction requested=false 2023-04-13 20:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:24,554 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:24,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:24,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,556 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:55:24,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:24,558 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:24,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@234b54cb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:24,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:24,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:24,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:24,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:24,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:24,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:24,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:24,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:24,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:24,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:24,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:24,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:24,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:24,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,573 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:24,573 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:24,575 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413684704d3dfee4b80926411114c30c087 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413684704d3dfee4b80926411114c30c087 2023-04-13 20:55:24,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419384591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419384595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419384599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,674 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:24,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:24,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:24,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:24,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:24,677 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,679 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:24,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:24,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:24,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:24,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:24,680 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:24,680 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:24,681 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:24,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@542ea77f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:24,685 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,685 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:24,686 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:24,687 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,687 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:24,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:24,689 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:24,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:24,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:24,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:24,690 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:24,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419384700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,706 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:24,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419384702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,706 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:24,706 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:24,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419384707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,807 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:24,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:24,810 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:24,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:24,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:24,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:24,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:24,814 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,815 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,815 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:24,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e0ecc26[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:24,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:24,817 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,819 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:24,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:24,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:24,820 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:24,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:24,824 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:24,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:24,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:24,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:24,824 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:24,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:24,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:24,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,830 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:24,830 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:24,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,830 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419384905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419384909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419384911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,930 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:24,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:24,933 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:24,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:24,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:24,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,933 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,936 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,936 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:24,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:24,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:24,937 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:24,937 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,937 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:24,937 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:24,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:24,939 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,939 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,939 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:24,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9e07c3f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:24,941 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:24,941 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,942 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:24,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,942 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:24,944 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:24,946 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:24,946 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:24,946 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:24,946 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:24,946 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:24,949 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,949 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:24,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,949 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:24,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:24,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:24,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:24,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:24,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:24,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:24,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:24,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:24,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:24,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:24,954 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:24,954 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:24,979 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/c0a3b82ce2ef41bd98e83a125e20c9b3 2023-04-13 20:55:25,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419385035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,040 DEBUG [Thread-2640] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:25,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419385037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,044 DEBUG [Thread-2642] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:25,054 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,058 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,062 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,062 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,062 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,063 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f49cc4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,066 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,066 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,068 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,071 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,071 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,071 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,073 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,081 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,082 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,081 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,182 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,185 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,186 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,188 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,188 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,190 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,190 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5140c8fa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,193 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,197 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,198 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,199 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,199 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,199 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,200 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,203 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,203 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:25,205 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,205 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419385211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419385213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419385217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,305 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,309 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,312 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,312 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,312 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,312 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79875108[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,317 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,317 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,318 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:25,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,322 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,322 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,329 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,329 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,330 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,329 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,330 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/6ff682b45355440ba98cdb4f04242903 2023-04-13 20:55:25,430 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,440 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,441 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,441 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,443 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,448 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,448 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,448 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,448 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,450 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738ccb83[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,453 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,453 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,454 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,454 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,454 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,457 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,460 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,460 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,460 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,460 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,460 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,462 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:25,468 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,469 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,469 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,569 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,572 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,575 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,577 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,577 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,577 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,577 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,579 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ca7b8a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,581 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,582 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,585 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,586 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,590 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,593 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,593 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,593 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,693 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,696 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,697 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,699 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,699 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,700 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:25,700 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:25,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:25,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:25,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:25,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,705 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@709a7091[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:25,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:25,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:25,710 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:25,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:25,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,711 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:25,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:25,714 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:25,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:25,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:25,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:25,714 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:25,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:25,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:25,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:25,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:25,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:25,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:25,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:25,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,720 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:25,721 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:25,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:25,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:25,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:25,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:25,721 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419385719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51164 deadline: 1681419385720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,726 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:25,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419385722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/5c8c06a59205488f9ac0996522757f1e 2023-04-13 20:55:25,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/c0a3b82ce2ef41bd98e83a125e20c9b3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3 2023-04-13 20:55:25,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3, entries=250, sequenceid=220, filesize=38.9 K 2023-04-13 20:55:25,808 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/6ff682b45355440ba98cdb4f04242903 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903 2023-04-13 20:55:25,811 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903, entries=150, sequenceid=220, filesize=11.8 K 2023-04-13 20:55:25,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/5c8c06a59205488f9ac0996522757f1e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e 2023-04-13 20:55:25,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e, entries=150, sequenceid=220, filesize=11.8 K 2023-04-13 20:55:25,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1259ms, sequenceid=220, compaction requested=true 2023-04-13 20:55:25,816 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:25,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:25,816 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:25,816 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:25,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:25,816 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91912 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:25,817 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:25,817 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b7a8d844cb464042a8c73d473eadecfd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=89.8 K 2023-04-13 20:55:25,817 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,817 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dcef5e7896804e72a857d5bbf4f4629e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.5 K 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b7a8d844cb464042a8c73d473eadecfd, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322050 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting dcef5e7896804e72a857d5bbf4f4629e, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322050 2023-04-13 20:55:25,817 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fa309cbc1e0a453e819391a245f6755d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681419322775 2023-04-13 20:55:25,818 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2a71bd48d6fd4323bca35f27a6a06cee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681419322775 2023-04-13 20:55:25,818 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c0a3b82ce2ef41bd98e83a125e20c9b3, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323912 2023-04-13 20:55:25,818 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6ff682b45355440ba98cdb4f04242903, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323927 2023-04-13 20:55:25,821 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:25,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:25,825 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#377 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:25,826 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:25,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:25,826 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:25,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:25,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:25,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:25,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:25,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:25,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:25,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:25,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:25,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:25,836 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:25,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:25,837 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:25,837 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:25,837 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:25,837 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:25,837 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#376 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:25,841 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/e5b993e8a7eb4efa8cb9588bfcf62706 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e5b993e8a7eb4efa8cb9588bfcf62706 2023-04-13 20:55:25,845 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into e5b993e8a7eb4efa8cb9588bfcf62706(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:25,845 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:25,845 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419325816; duration=0sec 2023-04-13 20:55:25,845 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:25,845 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:25,846 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:25,846 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:25,846 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:25,846 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e2227fc57f74697973dd4072ac621b3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.5 K 2023-04-13 20:55:25,846 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0e2227fc57f74697973dd4072ac621b3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1681419322050 2023-04-13 20:55:25,847 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cf04e3775d87467598bcad76a5859593, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1681419322775 2023-04-13 20:55:25,848 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5c8c06a59205488f9ac0996522757f1e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323927 2023-04-13 20:55:25,860 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#379 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:25,879 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134200b3ffae2f44ea808cc6acb8289898 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134200b3ffae2f44ea808cc6acb8289898 2023-04-13 20:55:25,890 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/69fc854ae3884a2488937978ccdbbfb8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/69fc854ae3884a2488937978ccdbbfb8 2023-04-13 20:55:25,895 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into 69fc854ae3884a2488937978ccdbbfb8(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:25,895 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:25,895 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419325816; duration=0sec 2023-04-13 20:55:25,895 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:25,900 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/acef846319d84f5c9d88ab8f38015aca 2023-04-13 20:55:26,292 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/5dd34d13328f45118c63c2ffe11b8d9a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5dd34d13328f45118c63c2ffe11b8d9a 2023-04-13 20:55:26,296 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into 5dd34d13328f45118c63c2ffe11b8d9a(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:26,296 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:26,296 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419325816; duration=0sec 2023-04-13 20:55:26,296 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:26,310 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/a3fe0ac8ab874a828df1292a7e60098b 2023-04-13 20:55:26,723 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/a5a8567cf6cf431e83c5e151d51bf980 2023-04-13 20:55:26,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:26,729 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:26,730 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/acef846319d84f5c9d88ab8f38015aca as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca 2023-04-13 20:55:26,733 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca, entries=150, sequenceid=239, filesize=25.3 K 2023-04-13 20:55:26,734 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/a3fe0ac8ab874a828df1292a7e60098b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b 2023-04-13 20:55:26,753 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b, entries=150, sequenceid=239, filesize=11.8 K 2023-04-13 20:55:26,754 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/a5a8567cf6cf431e83c5e151d51bf980 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980 2023-04-13 20:55:26,759 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980, entries=150, sequenceid=239, filesize=11.8 K 2023-04-13 20:55:26,759 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=46.96 KB/48090 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 923ms, sequenceid=239, compaction requested=false 2023-04-13 20:55:26,759 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:26,759 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:26,759 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:26,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:26,760 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:26,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:26,761 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,761 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,761 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:26,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:26,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:26,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,762 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:26,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:26,762 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:26,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3356709b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:26,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:26,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,765 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:26,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:26,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:26,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:26,767 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:26,767 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:26,771 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:26,771 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:26,771 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:26,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,774 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:26,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:26,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,778 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:26,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:26,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:26,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:26,788 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:26,788 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,788 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51150 deadline: 1681419386806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419386807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419386807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,888 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:26,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:26,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:26,892 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,892 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:26,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:26,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:26,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:26,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:26,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:26,896 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:26,896 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:26,901 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,902 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@261375a1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:26,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:26,905 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,906 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:26,906 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:26,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,906 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:26,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:26,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,908 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:26,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:26,910 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:26,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:26,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:26,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:26,910 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:26,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:26,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:26,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:26,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:26,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:26,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:26,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:26,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:26,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:26,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:26,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:26,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:26,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:26,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:26,919 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:26,919 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:26,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:26,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:26,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419386913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419386914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419386914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,019 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:27,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:27,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:27,022 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,022 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,023 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:27,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:27,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:27,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:27,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:27,025 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,025 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:27,025 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:27,025 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:27,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:27,027 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:27,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f15cc2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:27,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:27,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,030 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:27,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,030 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:27,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,033 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:27,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:27,036 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:27,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:27,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:27,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:27,036 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:27,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:27,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:27,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:27,043 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:27,044 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:27,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:27,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:27,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419387120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419387122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419387124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,144 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:27,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:27,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:27,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:27,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:27,148 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:27,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:27,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:27,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:27,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:27,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:27,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:27,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:27,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:27,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:27,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:27,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:27,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:27,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ccb0f5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:27,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:27,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:27,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:27,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:27,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:27,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:27,167 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:27,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:27,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:27,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:27,167 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:27,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:27,171 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:27,175 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:27,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:27,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:27,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:27,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:27,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:27,176 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:27,176 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:27,211 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413cd6d54aadde54b8abb656aa6ce9ab55f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cd6d54aadde54b8abb656aa6ce9ab55f 2023-04-13 20:55:27,218 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/e630abbca4cd49029ab95fa7e2a87649 2023-04-13 20:55:27,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/fbb82b3cc466494aad0bbbcc73e27041 2023-04-13 20:55:27,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/8734be5a9b23409e9f384e4837db27d0 2023-04-13 20:55:27,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/e630abbca4cd49029ab95fa7e2a87649 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649 2023-04-13 20:55:27,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649, entries=450, sequenceid=254, filesize=66.3 K 2023-04-13 20:55:27,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/fbb82b3cc466494aad0bbbcc73e27041 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041 2023-04-13 20:55:27,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041, entries=150, sequenceid=254, filesize=11.8 K 2023-04-13 20:55:27,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/8734be5a9b23409e9f384e4837db27d0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0 2023-04-13 20:55:27,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0, entries=150, sequenceid=254, filesize=11.8 K 2023-04-13 20:55:27,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 491ms, sequenceid=254, compaction requested=true 2023-04-13 20:55:27,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:27,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:27,251 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:27,251 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:27,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:27,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:27,252 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 119978 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:27,252 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:27,252 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:27,252 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:27,253 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,253 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,253 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5dd34d13328f45118c63c2ffe11b8d9a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=117.2 K 2023-04-13 20:55:27,253 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e5b993e8a7eb4efa8cb9588bfcf62706, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.6 K 2023-04-13 20:55:27,253 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5dd34d13328f45118c63c2ffe11b8d9a, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323927 2023-04-13 20:55:27,253 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e5b993e8a7eb4efa8cb9588bfcf62706, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323927 2023-04-13 20:55:27,253 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting acef846319d84f5c9d88ab8f38015aca, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681419324582 2023-04-13 20:55:27,253 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a3fe0ac8ab874a828df1292a7e60098b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681419324582 2023-04-13 20:55:27,254 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e630abbca4cd49029ab95fa7e2a87649, keycount=450, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326728 2023-04-13 20:55:27,254 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fbb82b3cc466494aad0bbbcc73e27041, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326744 2023-04-13 20:55:27,261 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#386 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:27,264 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#385 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:27,276 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:27,276 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/78fc10e1d0854d83823404f2cacff3ca as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/78fc10e1d0854d83823404f2cacff3ca 2023-04-13 20:55:27,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:27,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:27,279 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:27,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:27,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:27,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,281 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into 78fc10e1d0854d83823404f2cacff3ca(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:27,281 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:27,281 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419327251; duration=0sec 2023-04-13 20:55:27,281 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:27,281 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:27,281 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:27,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,282 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:27,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:27,282 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:27,282 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:27,282 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/69fc854ae3884a2488937978ccdbbfb8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=35.6 K 2023-04-13 20:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:27,283 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 69fc854ae3884a2488937978ccdbbfb8, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1681419323927 2023-04-13 20:55:27,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:27,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:27,283 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a5a8567cf6cf431e83c5e151d51bf980, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1681419324582 2023-04-13 20:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:27,283 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8734be5a9b23409e9f384e4837db27d0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326744 2023-04-13 20:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:27,283 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:27,283 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:27,284 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:27,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:27,298 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:27,312 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/d4d6733793e444fcb4320d3a1c89db59 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/d4d6733793e444fcb4320d3a1c89db59 2023-04-13 20:55:27,315 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into d4d6733793e444fcb4320d3a1c89db59(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:27,315 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:27,315 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419327251; duration=0sec 2023-04-13 20:55:27,315 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:27,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:27,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:27,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419387447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419387448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419387450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419387552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419387554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419387554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,713 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041343d6f92a13e243a9a2e67ae4cf68fd10 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041343d6f92a13e243a9a2e67ae4cf68fd10 2023-04-13 20:55:27,727 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/2ac484e36d6b42aca63241df6a3da7d8 2023-04-13 20:55:27,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419387757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419387758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:27,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:27,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419387758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419388062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419388062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419388063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,145 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/8e849eb1a19547b093d35e582a6d5536 2023-04-13 20:55:28,499 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/75847d68ae4d46e8b7f7472891a20b71 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/75847d68ae4d46e8b7f7472891a20b71 2023-04-13 20:55:28,503 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into 75847d68ae4d46e8b7f7472891a20b71(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 20:55:28,503 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:28,503 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419327251; duration=1sec 2023-04-13 20:55:28,503 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:28,558 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/6fa4a2e7202f458a8ecfabbc3ac22fe1 2023-04-13 20:55:28,562 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/2ac484e36d6b42aca63241df6a3da7d8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8 2023-04-13 20:55:28,566 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8, entries=150, sequenceid=275, filesize=25.4 K 2023-04-13 20:55:28,567 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/8e849eb1a19547b093d35e582a6d5536 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536 2023-04-13 20:55:28,570 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536, entries=150, sequenceid=275, filesize=11.9 K 2023-04-13 20:55:28,571 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/6fa4a2e7202f458a8ecfabbc3ac22fe1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1 2023-04-13 20:55:28,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419388567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419388570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419388571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,574 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1, entries=150, sequenceid=275, filesize=11.9 K 2023-04-13 20:55:28,575 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1291ms, sequenceid=275, compaction requested=false 2023-04-13 20:55:28,575 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:28,575 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:28,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:28,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:28,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:28,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:28,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:28,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@719ec4ff[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:28,578 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:28,578 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:28,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:28,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:28,586 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:28,589 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:28,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:28,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:28,589 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:28,589 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:28,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:28,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:28,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:28,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:28,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,596 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:28,596 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:28,696 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:28,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:28,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:28,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:28,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:28,703 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:28,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:28,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:28,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:28,704 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,704 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:28,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:28,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:28,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:28,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:28,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:28,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:28,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:28,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:28,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:28,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:28,712 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:28,712 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:28,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:28,749 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d3c59177f38244ec8ee7f0d761057ffe to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3c59177f38244ec8ee7f0d761057ffe 2023-04-13 20:55:29,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:29,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:29,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419389117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419389119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,154 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=289, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/1d99d6c7614a4da2a058709168903413 2023-04-13 20:55:29,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419389224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419389225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419389427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419389427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,566 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/c88da419f6b3403eac238d7c17d89780 2023-04-13 20:55:29,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419389576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419389578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419389582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419389734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:29,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419389734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:29,987 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=289 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/39b612a1db0e486296fd5349867805f9 2023-04-13 20:55:29,991 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/1d99d6c7614a4da2a058709168903413 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413 2023-04-13 20:55:29,994 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413, entries=150, sequenceid=289, filesize=25.4 K 2023-04-13 20:55:29,995 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/c88da419f6b3403eac238d7c17d89780 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780 2023-04-13 20:55:29,998 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780, entries=150, sequenceid=289, filesize=11.9 K 2023-04-13 20:55:29,999 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/39b612a1db0e486296fd5349867805f9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9 2023-04-13 20:55:30,002 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9, entries=150, sequenceid=289, filesize=11.9 K 2023-04-13 20:55:30,002 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1290ms, sequenceid=289, compaction requested=true 2023-04-13 20:55:30,003 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:30,003 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:30,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:30,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,004 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,005 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:30,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c1d1910[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:30,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:30,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,008 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:30,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:30,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:30,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:30,010 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:30,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:30,013 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:30,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:30,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:30,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:30,013 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:30,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:30,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:30,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:30,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:30,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:30,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:30,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:30,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,019 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,019 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:30,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:30,019 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:30,019 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:30,119 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:30,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:30,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:30,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:30,122 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:30,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:30,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:30,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:30,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:30,124 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:30,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:30,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:30,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:30,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:30,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:30,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:30,125 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:30,125 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:30,125 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:30,126 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:30,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:30,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:30,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419390272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419390273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419390380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419390380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,537 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304132d6b6c6a228849d88bae969848394cba to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132d6b6c6a228849d88bae969848394cba 2023-04-13 20:55:30,543 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/bd173774d4c44cd3b9e78d8385a2a457 2023-04-13 20:55:30,556 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/518dd7563b5742d9b7a1ce6181f9bd57 2023-04-13 20:55:30,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419390587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419390587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419390898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:30,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419390899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,968 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/2cc781807b774a918ebed8bb8a8f47f7 2023-04-13 20:55:30,971 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/bd173774d4c44cd3b9e78d8385a2a457 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457 2023-04-13 20:55:30,976 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457, entries=150, sequenceid=309, filesize=25.4 K 2023-04-13 20:55:30,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/518dd7563b5742d9b7a1ce6181f9bd57 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57 2023-04-13 20:55:30,981 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57, entries=150, sequenceid=309, filesize=11.9 K 2023-04-13 20:55:30,981 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/2cc781807b774a918ebed8bb8a8f47f7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7 2023-04-13 20:55:30,984 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7, entries=150, sequenceid=309, filesize=11.9 K 2023-04-13 20:55:30,985 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 860ms, sequenceid=309, compaction requested=true 2023-04-13 20:55:30,985 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:30,985 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,987 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:30,987 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:30,988 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:30,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:30,988 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d25b1d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:30,990 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:30,990 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,992 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:30,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:30,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:30,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:30,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:30,994 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:30,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:30,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:30,994 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:30,994 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:30,995 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,996 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:30,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,996 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:30,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:30,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,997 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:30,998 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:31,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,004 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,005 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:31,005 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:31,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:31,105 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:31,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:31,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:31,108 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:31,110 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:31,110 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,110 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:31,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:31,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:31,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:31,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:31,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:31,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:31,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:31,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:31,111 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:31,111 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:31,112 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:31,112 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:31,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:31,410 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:31,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419391452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419391456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,524 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137c9541f26c6d49b58a4dbb7a1de4b8b4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137c9541f26c6d49b58a4dbb7a1de4b8b4 2023-04-13 20:55:31,530 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9be24837c86446d8b3975afaa33a355d 2023-04-13 20:55:31,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419391564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419391566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419391584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,594 DEBUG [Thread-2646] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:31,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419391600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,605 DEBUG [Thread-2644] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:31,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419391604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,612 DEBUG [Thread-2638] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:31,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419391771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:31,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419391772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,941 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/5d93d29438b34d02bf61417ca8ef0c7f 2023-04-13 20:55:31,951 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b8a0dfbc10fb44db98153a16fafc430e 2023-04-13 20:55:31,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9be24837c86446d8b3975afaa33a355d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d 2023-04-13 20:55:31,959 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d, entries=150, sequenceid=321, filesize=25.4 K 2023-04-13 20:55:31,959 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/5d93d29438b34d02bf61417ca8ef0c7f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f 2023-04-13 20:55:31,964 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f, entries=150, sequenceid=321, filesize=11.9 K 2023-04-13 20:55:31,964 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b8a0dfbc10fb44db98153a16fafc430e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e 2023-04-13 20:55:31,968 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e, entries=150, sequenceid=321, filesize=11.9 K 2023-04-13 20:55:31,969 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 857ms, sequenceid=321, compaction requested=true 2023-04-13 20:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:31,969 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:31,969 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:31,971 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:31,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:31,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:31,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8d0a6a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:31,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:31,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:31,975 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:31,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:31,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:31,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:31,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:31,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:31,987 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:31,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:31,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:31,988 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:31,988 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:31,989 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,989 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:31,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,989 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:31,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,990 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,991 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:31,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:31,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:31,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:31,993 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:31,993 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:31,993 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:31,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:31,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,090 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-13 20:55:32,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:32,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:32,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:32,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:32,091 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:32,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419331447, max timestamp=1681419332089Num uniques 859; 2023-04-13 20:55:32,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:32,091 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:32,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:32,093 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,098 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,102 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,102 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,106 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,106 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,106 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,106 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,108 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304134d57b81343d9439ab5858b12af32d643 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134d57b81343d9439ab5858b12af32d643 2023-04-13 20:55:32,108 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,108 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7338fb8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,112 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,112 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,113 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,115 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,116 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,116 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,116 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,119 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,127 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,134 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,134 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,134 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,135 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,135 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,135 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419392146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419392153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,235 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,240 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,241 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,242 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,242 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,244 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,246 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,247 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c2c3d7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,250 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,257 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,257 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,257 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,259 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,259 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419392254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,264 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419392257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,265 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,265 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,265 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,265 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,365 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,369 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,370 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,370 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,371 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,372 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,372 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,372 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,372 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,374 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,374 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@254e7e01[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,376 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,378 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,378 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,379 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,379 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,383 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,383 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,383 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,384 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,392 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,392 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,393 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,393 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419392463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419392465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,493 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,496 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,496 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,498 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,498 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,498 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,500 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,502 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,502 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df8ee98[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,504 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,504 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,505 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,505 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,507 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,507 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,508 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,511 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,511 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,513 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,513 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,513 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,515 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,516 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=342, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/3cc8969abc7e49e5904fa2dba44c82d1 2023-04-13 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,517 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,518 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,519 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,519 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,519 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,619 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,622 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,622 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,622 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,622 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,622 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,622 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,624 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,625 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,625 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,625 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,625 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,625 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,626 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,626 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,628 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@562df6d0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,631 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,631 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,632 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,632 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,635 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,635 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,635 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,635 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,635 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,638 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,641 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,641 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,641 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,742 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,745 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,745 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,745 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,745 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,748 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,751 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,751 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,752 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,752 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,754 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63c45bb9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,756 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,757 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,757 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,760 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,760 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,765 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,765 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,765 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,766 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,767 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,768 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,768 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,768 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,775 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,775 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419392770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,776 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,776 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:32,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51162 deadline: 1681419392770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,876 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:32,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:32,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:32,879 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:32,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:32,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:32,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,881 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,881 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:32,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:32,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:32,882 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:32,882 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:32,882 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:32,882 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:32,882 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:32,882 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,882 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:32,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:32,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:32,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:32,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:32,883 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:32,885 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ac9fb6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:32,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:32,887 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,889 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:32,889 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:32,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:32,891 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:32,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:32,893 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:32,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:32,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:32,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:32,893 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:32,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:32,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:32,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:32,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:32,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:32,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:32,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:32,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:32,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:32,899 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,900 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:32,900 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:32,900 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,900 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:32,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:32,900 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:32,900 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:32,900 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:32,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:32,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:32,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/38cf3dc928fb4a7fba3061f4c525a352 2023-04-13 20:55:32,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/792e561296a04b3c84c3cf55d595036f 2023-04-13 20:55:32,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/3cc8969abc7e49e5904fa2dba44c82d1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1 2023-04-13 20:55:32,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1, entries=200, sequenceid=342, filesize=32.3 K 2023-04-13 20:55:32,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/38cf3dc928fb4a7fba3061f4c525a352 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352 2023-04-13 20:55:32,952 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352, entries=150, sequenceid=342, filesize=11.9 K 2023-04-13 20:55:32,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/792e561296a04b3c84c3cf55d595036f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f 2023-04-13 20:55:32,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f, entries=150, sequenceid=342, filesize=11.9 K 2023-04-13 20:55:32,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=87.22 KB/89310 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 869ms, sequenceid=342, compaction requested=true 2023-04-13 20:55:32,960 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:32,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:32,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:55:32,960 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:55:32,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:32,960 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:32,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73410 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:55:32,962 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:32,962 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,962 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/78fc10e1d0854d83823404f2cacff3ca, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=71.7 K 2023-04-13 20:55:32,962 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 78fc10e1d0854d83823404f2cacff3ca, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326744 2023-04-13 20:55:32,962 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8e849eb1a19547b093d35e582a6d5536, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681419326787 2023-04-13 20:55:32,963 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c88da419f6b3403eac238d7c17d89780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681419327431 2023-04-13 20:55:32,963 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 163610 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:55:32,963 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 518dd7563b5742d9b7a1ce6181f9bd57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419329108 2023-04-13 20:55:32,963 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:32,963 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:32,963 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/75847d68ae4d46e8b7f7472891a20b71, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=159.8 K 2023-04-13 20:55:32,963 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5d93d29438b34d02bf61417ca8ef0c7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681419330260 2023-04-13 20:55:32,964 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 75847d68ae4d46e8b7f7472891a20b71, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326744 2023-04-13 20:55:32,964 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 38cf3dc928fb4a7fba3061f4c525a352, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:32,964 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2ac484e36d6b42aca63241df6a3da7d8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681419326787 2023-04-13 20:55:32,964 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1d99d6c7614a4da2a058709168903413, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681419327431 2023-04-13 20:55:32,965 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting bd173774d4c44cd3b9e78d8385a2a457, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419329108 2023-04-13 20:55:32,965 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9be24837c86446d8b3975afaa33a355d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681419330260 2023-04-13 20:55:32,965 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 3cc8969abc7e49e5904fa2dba44c82d1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:32,974 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#403 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:32,981 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#404 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:33,000 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:33,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:33,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:33,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:33,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:33,003 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:33,003 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:33,003 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:33,003 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:33,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,004 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:33,005 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:33,005 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:33,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:33,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:33,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:33,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:33,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:33,008 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:33,009 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:33,010 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:33,010 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:33,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:33,284 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:33,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51212 deadline: 1681419393346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419393351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,387 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/3debd07b4fe444a6a4ca6ad1cdcc985b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/3debd07b4fe444a6a4ca6ad1cdcc985b 2023-04-13 20:55:33,391 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into 3debd07b4fe444a6a4ca6ad1cdcc985b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:33,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:33,391 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=10, startTime=1681419332960; duration=0sec 2023-04-13 20:55:33,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:33,391 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:55:33,394 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73410 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:55:33,394 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:33,394 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:33,394 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/d4d6733793e444fcb4320d3a1c89db59, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=71.7 K 2023-04-13 20:55:33,395 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting d4d6733793e444fcb4320d3a1c89db59, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1681419326744 2023-04-13 20:55:33,395 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6fa4a2e7202f458a8ecfabbc3ac22fe1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1681419326787 2023-04-13 20:55:33,395 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 39b612a1db0e486296fd5349867805f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=289, earliestPutTs=1681419327431 2023-04-13 20:55:33,397 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2cc781807b774a918ebed8bb8a8f47f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1681419329108 2023-04-13 20:55:33,397 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b8a0dfbc10fb44db98153a16fafc430e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1681419330260 2023-04-13 20:55:33,398 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 792e561296a04b3c84c3cf55d595036f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:33,414 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#406 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:33,434 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413ae7b753831674536955811333d3e8f46 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ae7b753831674536955811333d3e8f46 2023-04-13 20:55:33,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419393460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419393461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419393666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419393666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,802 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/f926bc4948044d5e8c8d9c4166c195cc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/f926bc4948044d5e8c8d9c4166c195cc 2023-04-13 20:55:33,808 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into f926bc4948044d5e8c8d9c4166c195cc(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:33,808 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:33,808 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=10, startTime=1681419332960; duration=0sec 2023-04-13 20:55:33,808 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:33,824 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/f1d937c4bd214f119eea7bb6dafb07b0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f1d937c4bd214f119eea7bb6dafb07b0 2023-04-13 20:55:33,829 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into f1d937c4bd214f119eea7bb6dafb07b0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:33,829 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:33,829 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=10, startTime=1681419332960; duration=0sec 2023-04-13 20:55:33,829 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:33,843 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=358, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/5a36025a472e44fe8c132b76d0810d31 2023-04-13 20:55:33,861 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/dffe25fdf1c24cb6915a4c2663b9e1bf 2023-04-13 20:55:33,872 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b0f7046ee8c4459a8f406d050c889223 2023-04-13 20:55:33,875 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/5a36025a472e44fe8c132b76d0810d31 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31 2023-04-13 20:55:33,878 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31, entries=150, sequenceid=358, filesize=25.4 K 2023-04-13 20:55:33,879 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/dffe25fdf1c24cb6915a4c2663b9e1bf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf 2023-04-13 20:55:33,882 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf, entries=150, sequenceid=358, filesize=11.9 K 2023-04-13 20:55:33,883 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/b0f7046ee8c4459a8f406d050c889223 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223 2023-04-13 20:55:33,886 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223, entries=150, sequenceid=358, filesize=11.9 K 2023-04-13 20:55:33,887 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 877ms, sequenceid=358, compaction requested=false 2023-04-13 20:55:33,887 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:33,887 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:33,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:33,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:33,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:33,888 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:33,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:33,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:33,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f2a2bb5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:33,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:33,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:33,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,894 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:33,895 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:33,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:33,895 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:33,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:33,899 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:33,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:33,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:33,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:33,899 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:33,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:33,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:33,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:33,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:33,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:33,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:33,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:33,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:33,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:33,907 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:33,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:33,907 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:33,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:33,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:33,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:34,008 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419394005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,011 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419394009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,013 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,013 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,014 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,015 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,015 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,015 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,015 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,015 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,016 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,017 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,017 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,017 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,017 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688d1f6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,019 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,021 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,021 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,021 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,022 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,022 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,024 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,025 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,028 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,032 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,033 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,034 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,034 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419394110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419394112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,134 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,137 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,137 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,144 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,145 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,147 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,147 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d04b0d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,149 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,151 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,153 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,153 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,160 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,161 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,164 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,165 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,164 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:34,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,265 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,268 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,272 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,272 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,274 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,274 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16502d6e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,279 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,279 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,280 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,282 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,283 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,284 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:34,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,289 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,289 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419394317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,319 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419394317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,389 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,396 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,399 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,399 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,400 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,400 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,401 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,401 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,401 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041313bf2466df614dbfb3eb34908fc4a563 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313bf2466df614dbfb3eb34908fc4a563 2023-04-13 20:55:34,403 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,403 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c9db92[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,408 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,409 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,410 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=378, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/d21ffe04ca6e4e08a83daabbf2c3cc90 2023-04-13 20:55:34,410 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,411 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,414 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,415 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,421 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/af920f80c1884613985988dfdeee16d7 2023-04-13 20:55:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,428 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,428 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,429 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,429 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,529 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,532 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,534 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,536 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,537 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,538 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@380ea21c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,550 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,554 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,554 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,555 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,559 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,559 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419394622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:34,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419394622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,660 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,662 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,663 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,663 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,665 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,665 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,666 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,667 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,667 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,667 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc30e33[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,672 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,672 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,675 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,675 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,680 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:34,681 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,681 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,681 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,781 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,784 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,784 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,784 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,784 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,785 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,786 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,787 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,787 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,787 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:34,787 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:34,788 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:34,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,789 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:34,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6f67b8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:34,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:34,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:34,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:34,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:34,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:34,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:34,796 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,797 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:34,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:34,799 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:34,799 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:34,799 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:34,799 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:34,799 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:34,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,801 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:34,802 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:34,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:34,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:34,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:34,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:34,804 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:34,804 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:34,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:34,842 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/de80f91aae4d4d6da5d6222647df2738 2023-04-13 20:55:34,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/d21ffe04ca6e4e08a83daabbf2c3cc90 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90 2023-04-13 20:55:34,850 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90, entries=200, sequenceid=378, filesize=32.3 K 2023-04-13 20:55:34,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/af920f80c1884613985988dfdeee16d7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7 2023-04-13 20:55:34,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7, entries=150, sequenceid=378, filesize=11.9 K 2023-04-13 20:55:34,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/de80f91aae4d4d6da5d6222647df2738 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738 2023-04-13 20:55:34,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738, entries=150, sequenceid=378, filesize=11.9 K 2023-04-13 20:55:34,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 882ms, sequenceid=378, compaction requested=true 2023-04-13 20:55:34,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:34,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:34,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:55:34,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:55:34,859 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:34,859 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:34,860 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:34,860 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:34,860 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,860 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/f926bc4948044d5e8c8d9c4166c195cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=83.8 K 2023-04-13 20:55:34,861 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f926bc4948044d5e8c8d9c4166c195cc, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:34,861 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:34,861 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:34,861 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,861 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f1d937c4bd214f119eea7bb6dafb07b0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.3 K 2023-04-13 20:55:34,862 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5a36025a472e44fe8c132b76d0810d31, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681419332116 2023-04-13 20:55:34,862 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f1d937c4bd214f119eea7bb6dafb07b0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:34,862 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d21ffe04ca6e4e08a83daabbf2c3cc90, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:34,862 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b0f7046ee8c4459a8f406d050c889223, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681419332116 2023-04-13 20:55:34,862 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting de80f91aae4d4d6da5d6222647df2738, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:34,867 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:34,871 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#412 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:34,904 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:34,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:34,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:34,907 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:34,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:34,907 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:34,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:34,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:34,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:34,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:34,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:34,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:34,912 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:35,134 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:35,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419395163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419395167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419395268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419395285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,290 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/62bb14dd07f24b8f904834a56a09397e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/62bb14dd07f24b8f904834a56a09397e 2023-04-13 20:55:35,295 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into 62bb14dd07f24b8f904834a56a09397e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:35,295 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:35,295 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419334859; duration=0sec 2023-04-13 20:55:35,295 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:35,295 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:35,296 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:35,296 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:35,296 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:35,296 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/3debd07b4fe444a6a4ca6ad1cdcc985b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.3 K 2023-04-13 20:55:35,297 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3debd07b4fe444a6a4ca6ad1cdcc985b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1681419331447 2023-04-13 20:55:35,297 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting dffe25fdf1c24cb6915a4c2663b9e1bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1681419332116 2023-04-13 20:55:35,297 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting af920f80c1884613985988dfdeee16d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:35,301 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#415 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:35,331 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041378ac8c9c02834a9c8a62b12d9c910349 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041378ac8c9c02834a9c8a62b12d9c910349 2023-04-13 20:55:35,340 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=393, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/28b96aabd8f84656b30e8da1d7accac5 2023-04-13 20:55:35,348 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/2968a13292404c48b0706ad805258944 2023-04-13 20:55:35,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419395487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419395489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51184 deadline: 1681419395597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,603 DEBUG [Thread-2646] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:35,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51150 deadline: 1681419395628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,636 DEBUG [Thread-2644] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8184 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:35,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51164 deadline: 1681419395648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,655 DEBUG [Thread-2638] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8205 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:35,696 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/b49c581273304e8cae8fa68b70b49378 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b49c581273304e8cae8fa68b70b49378 2023-04-13 20:55:35,700 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into b49c581273304e8cae8fa68b70b49378(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:35,700 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:35,700 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419334859; duration=0sec 2023-04-13 20:55:35,700 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:35,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/ad23118ef9aa49719e4dde9840cd2966 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/ad23118ef9aa49719e4dde9840cd2966 2023-04-13 20:55:35,721 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into ad23118ef9aa49719e4dde9840cd2966(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:35,721 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:35,722 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419334859; duration=0sec 2023-04-13 20:55:35,722 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:35,756 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/9647e901b1714397bf026310978c1f60 2023-04-13 20:55:35,762 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/28b96aabd8f84656b30e8da1d7accac5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5 2023-04-13 20:55:35,767 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5, entries=150, sequenceid=393, filesize=25.4 K 2023-04-13 20:55:35,767 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/2968a13292404c48b0706ad805258944 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944 2023-04-13 20:55:35,770 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944, entries=150, sequenceid=393, filesize=11.9 K 2023-04-13 20:55:35,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/9647e901b1714397bf026310978c1f60 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60 2023-04-13 20:55:35,774 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60, entries=150, sequenceid=393, filesize=11.9 K 2023-04-13 20:55:35,774 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 862ms, sequenceid=393, compaction requested=false 2023-04-13 20:55:35,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:35,775 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:35,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:35,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:35,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:35,775 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:35,776 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,776 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,777 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:35,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ccc242c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,781 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:35,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:35,781 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:35,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,782 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:35,787 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:35,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:35,790 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:35,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:35,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:35,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:35,790 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:35,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:35,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:35,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:35,798 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:35,798 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:35,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:35,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:35,797 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,799 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,800 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:35,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:35,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:55:35,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:35,801 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:35,801 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:35,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:35,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:35,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:35,829 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413160e8965e9d8424799aa10cf1cac3795 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413160e8965e9d8424799aa10cf1cac3795 2023-04-13 20:55:35,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419395839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,845 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=414, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/63faa74ed12c4f0c902dff29764f85e4 2023-04-13 20:55:35,850 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419395844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,898 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:35,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:35,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:35,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:35,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:35,901 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:35,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:35,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:35,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:35,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:35,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:35,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:35,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:35,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:35,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:35,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:35,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:35,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:35,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:35,904 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:35,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,907 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:35,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@119f73eb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:35,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:35,908 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,909 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:35,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,910 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:35,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:35,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:35,911 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:35,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:35,913 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:35,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:35,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:35,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:35,913 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:35,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,915 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:35,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:35,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:35,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:35,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:35,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:35,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:35,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:35,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:35,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:35,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:35,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:35,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:35,919 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:35,919 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:35,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419395945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:35,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:35,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419395952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,019 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,024 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,024 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,026 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:36,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:36,028 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:36,030 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:36,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7448557d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:36,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:36,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:36,034 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:36,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:36,036 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:36,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,036 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:36,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:36,039 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:36,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:36,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:36,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:36,039 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:36,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,040 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,042 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:36,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:36,047 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:36,047 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:36,147 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,150 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,150 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,151 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419396151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,153 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:36,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:36,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:36,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:36,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658baeeb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:36,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:36,158 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:36,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:36,160 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:36,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:36,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:36,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419396159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,161 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,161 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:36,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:36,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:36,163 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:36,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:36,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:36,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:36,163 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:36,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,166 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,169 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:36,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,170 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,170 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:36,170 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:36,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/5c5a8deb60a74568a5451e7e35b8cd4c 2023-04-13 20:55:36,270 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,273 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,278 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,279 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,279 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,279 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,279 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:36,279 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:36,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:36,280 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:36,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:36,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,282 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:36,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@585b1b32[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:36,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:36,285 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:36,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:36,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:36,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,286 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:36,287 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,287 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:36,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,288 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:36,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:36,290 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:36,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:36,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:36,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:36,290 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:36,292 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:36,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,297 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:36,297 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:36,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,397 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,411 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,411 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,413 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,414 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:36,415 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:36,415 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:36,418 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,418 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:36,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:36,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32b12bf9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:36,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:36,422 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:36,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:36,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:36,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:36,425 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:36,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:36,430 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:36,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:36,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:36,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:36,430 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:36,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,443 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,443 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:36,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,446 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:36,446 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:36,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:36,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:36,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419396457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:36,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419396462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,546 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,566 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,569 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:36,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:36,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:36,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:36,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464736f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:36,593 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:36,593 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:36,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:36,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:36,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:36,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:36,600 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:36,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:36,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:36,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:36,601 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:36,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:36,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,604 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,619 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:36,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:36,619 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:36,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,619 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,619 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:36,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:36,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:36,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/fc50229c0f354817ba06af5611963be8 2023-04-13 20:55:36,673 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/63faa74ed12c4f0c902dff29764f85e4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4 2023-04-13 20:55:36,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4, entries=200, sequenceid=414, filesize=32.3 K 2023-04-13 20:55:36,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/5c5a8deb60a74568a5451e7e35b8cd4c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c 2023-04-13 20:55:36,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c, entries=150, sequenceid=414, filesize=11.9 K 2023-04-13 20:55:36,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/fc50229c0f354817ba06af5611963be8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8 2023-04-13 20:55:36,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8, entries=150, sequenceid=414, filesize=11.9 K 2023-04-13 20:55:36,684 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 883ms, sequenceid=414, compaction requested=true 2023-04-13 20:55:36,684 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:36,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:36,684 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:36,684 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:36,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:36,684 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:36,685 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85928 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:36,685 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:36,685 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,685 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b49c581273304e8cae8fa68b70b49378, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=83.9 K 2023-04-13 20:55:36,686 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:36,686 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:36,686 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,686 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/ad23118ef9aa49719e4dde9840cd2966, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.4 K 2023-04-13 20:55:36,686 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b49c581273304e8cae8fa68b70b49378, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:36,687 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ad23118ef9aa49719e4dde9840cd2966, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:36,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 28b96aabd8f84656b30e8da1d7accac5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681419333997 2023-04-13 20:55:36,687 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2968a13292404c48b0706ad805258944, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681419333997 2023-04-13 20:55:36,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 63faa74ed12c4f0c902dff29764f85e4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:36,687 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5c5a8deb60a74568a5451e7e35b8cd4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:36,693 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:36,697 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#422 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:36,720 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:36,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:36,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:36,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:36,743 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:36,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:36,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:36,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,745 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:36,745 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:36,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:36,746 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:36,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:36,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:36,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:36,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:36,768 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:36,770 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:36,770 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,770 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. started... 2023-04-13 20:55:36,770 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:36,771 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:36,774 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/826d1988c4b3454d803be5f9117cc012 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/826d1988c4b3454d803be5f9117cc012 2023-04-13 20:55:36,780 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into 826d1988c4b3454d803be5f9117cc012(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:36,780 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:36,780 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419336684; duration=0sec 2023-04-13 20:55:36,780 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:36,780 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:36,781 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:36,781 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:36,781 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:36,781 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/62bb14dd07f24b8f904834a56a09397e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.4 K 2023-04-13 20:55:36,782 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 62bb14dd07f24b8f904834a56a09397e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1681419333332 2023-04-13 20:55:36,783 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9647e901b1714397bf026310978c1f60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1681419333997 2023-04-13 20:55:36,783 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fc50229c0f354817ba06af5611963be8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:36,805 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:36,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:36,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. as already flushing 2023-04-13 20:55:37,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419397023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419397028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,104 DEBUG [Thread-2649] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:54584 2023-04-13 20:55:37,104 DEBUG [Thread-2649] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:37,106 DEBUG [Thread-2657] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7eae4191 to 127.0.0.1:54584 2023-04-13 20:55:37,106 DEBUG [Thread-2657] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:37,106 DEBUG [Thread-2653] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:54584 2023-04-13 20:55:37,106 DEBUG [Thread-2653] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:37,110 DEBUG [Thread-2651] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:54584 2023-04-13 20:55:37,110 DEBUG [Thread-2651] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:37,110 DEBUG [Thread-2655] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:54584 2023-04-13 20:55:37,110 DEBUG [Thread-2655] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:37,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419397129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,136 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419397136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,194 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304135fd5e5988172463abde9f2e5ac647256 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135fd5e5988172463abde9f2e5ac647256 2023-04-13 20:55:37,217 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/bdf8121477d64927a2bbdd8d91a01278 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/bdf8121477d64927a2bbdd8d91a01278 2023-04-13 20:55:37,220 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into bdf8121477d64927a2bbdd8d91a01278(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:37,220 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:37,220 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419336684; duration=0sec 2023-04-13 20:55:37,221 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:37,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419397331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419397337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,588 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/d55c192d7d044a82a01c2ea782dfacc3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d55c192d7d044a82a01c2ea782dfacc3 2023-04-13 20:55:37,591 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into d55c192d7d044a82a01c2ea782dfacc3(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:37,591 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:37,591 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419336684; duration=0sec 2023-04-13 20:55:37,591 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:37,600 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=428, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9740e6bc74654e03a1075f8ed3e970a1 2023-04-13 20:55:37,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419397634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:37,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:37,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419397639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,008 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/8fad9ca2c381421bad0e238477f906ee 2023-04-13 20:55:38,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:38,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51162 deadline: 1681419398138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:38,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51212 deadline: 1681419398142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,416 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/f2bcefe635ad44b4b2ec3040d26a95f3 2023-04-13 20:55:38,420 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9740e6bc74654e03a1075f8ed3e970a1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1 2023-04-13 20:55:38,422 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1, entries=150, sequenceid=428, filesize=25.4 K 2023-04-13 20:55:38,423 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/8fad9ca2c381421bad0e238477f906ee as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee 2023-04-13 20:55:38,426 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee, entries=150, sequenceid=428, filesize=11.9 K 2023-04-13 20:55:38,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/f2bcefe635ad44b4b2ec3040d26a95f3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3 2023-04-13 20:55:38,429 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3, entries=150, sequenceid=428, filesize=11.9 K 2023-04-13 20:55:38,430 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1660ms, sequenceid=428, compaction requested=false 2023-04-13 20:55:38,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:38,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:38,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:38,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,432 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:38,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:38,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c2079fe[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:38,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:38,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,435 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:38,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:38,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:38,436 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:38,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:38,438 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:38,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:38,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:38,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:38,438 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:38,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:38,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:38,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:38,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,441 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:38,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:38,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:38,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:38,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:38,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:38,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:38,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,444 DEBUG [Thread-2648] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:38,445 DEBUG [Thread-2648] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:38,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:38,445 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:38,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:38,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:38,545 DEBUG [Thread-2648] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:38,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:39,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:39,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:55:39,150 DEBUG [Thread-2640] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:54584 2023-04-13 20:55:39,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:39,151 DEBUG [Thread-2642] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:54584 2023-04-13 20:55:39,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:39,151 DEBUG [Thread-2642] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:39,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:39,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:39,150 DEBUG [Thread-2640] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:39,151 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:39,151 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:39,265 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 20:55:39,300 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/71cb3d037b7a439381fa023283f30d02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b7a8d844cb464042a8c73d473eadecfd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5dd34d13328f45118c63c2ffe11b8d9a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/75847d68ae4d46e8b7f7472891a20b71, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/f926bc4948044d5e8c8d9c4166c195cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b49c581273304e8cae8fa68b70b49378, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4] to archive 2023-04-13 20:55:39,301 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:39,303 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c3241d1fc5f143cfa29f2dc4178d2dbb 2023-04-13 20:55:39,305 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/4d408a4c60834717aa521a5f2505ea23 2023-04-13 20:55:39,306 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/068abe1c8df642a985045e3d4a1fa1cb 2023-04-13 20:55:39,308 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bceb93650cec499c84a147a2483dbc68 2023-04-13 20:55:39,309 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/71cb3d037b7a439381fa023283f30d02 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/71cb3d037b7a439381fa023283f30d02 2023-04-13 20:55:39,310 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5482cddb894248e8ab726478ea0a54d4 2023-04-13 20:55:39,311 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/054afbed8b014a799377ac19366244ea 2023-04-13 20:55:39,312 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b7a8d844cb464042a8c73d473eadecfd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b7a8d844cb464042a8c73d473eadecfd 2023-04-13 20:55:39,313 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/fa309cbc1e0a453e819391a245f6755d 2023-04-13 20:55:39,314 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/c0a3b82ce2ef41bd98e83a125e20c9b3 2023-04-13 20:55:39,316 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5dd34d13328f45118c63c2ffe11b8d9a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5dd34d13328f45118c63c2ffe11b8d9a 2023-04-13 20:55:39,317 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/acef846319d84f5c9d88ab8f38015aca 2023-04-13 20:55:39,319 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/e630abbca4cd49029ab95fa7e2a87649 2023-04-13 20:55:39,320 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/75847d68ae4d46e8b7f7472891a20b71 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/75847d68ae4d46e8b7f7472891a20b71 2023-04-13 20:55:39,321 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/2ac484e36d6b42aca63241df6a3da7d8 2023-04-13 20:55:39,323 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/1d99d6c7614a4da2a058709168903413 2023-04-13 20:55:39,324 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/bd173774d4c44cd3b9e78d8385a2a457 2023-04-13 20:55:39,324 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9be24837c86446d8b3975afaa33a355d 2023-04-13 20:55:39,325 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/3cc8969abc7e49e5904fa2dba44c82d1 2023-04-13 20:55:39,326 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/f926bc4948044d5e8c8d9c4166c195cc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/f926bc4948044d5e8c8d9c4166c195cc 2023-04-13 20:55:39,327 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/5a36025a472e44fe8c132b76d0810d31 2023-04-13 20:55:39,328 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d21ffe04ca6e4e08a83daabbf2c3cc90 2023-04-13 20:55:39,329 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b49c581273304e8cae8fa68b70b49378 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/b49c581273304e8cae8fa68b70b49378 2023-04-13 20:55:39,329 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/28b96aabd8f84656b30e8da1d7accac5 2023-04-13 20:55:39,330 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/63faa74ed12c4f0c902dff29764f85e4 2023-04-13 20:55:39,334 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e45bf15223de44d18caa0f9965a5b59d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dcef5e7896804e72a857d5bbf4f4629e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e5b993e8a7eb4efa8cb9588bfcf62706, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/78fc10e1d0854d83823404f2cacff3ca, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/3debd07b4fe444a6a4ca6ad1cdcc985b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/ad23118ef9aa49719e4dde9840cd2966, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c] to archive 2023-04-13 20:55:39,335 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:39,336 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/61834871f3724d8fb31909418c1bb5d6 2023-04-13 20:55:39,337 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/9fe857ffc4ab4e5b976576c95422d707 2023-04-13 20:55:39,338 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/de6647c591e34d3d841590523a7400a5 2023-04-13 20:55:39,339 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e45bf15223de44d18caa0f9965a5b59d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e45bf15223de44d18caa0f9965a5b59d 2023-04-13 20:55:39,339 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/31553bd408334ac0880e5bffad2a3979 2023-04-13 20:55:39,340 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/bd4a79904948438f90285b8cbe41bf11 2023-04-13 20:55:39,341 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dcef5e7896804e72a857d5bbf4f4629e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dcef5e7896804e72a857d5bbf4f4629e 2023-04-13 20:55:39,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/4654b16aa4014020bff4f570260cc3c6 2023-04-13 20:55:39,342 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2a71bd48d6fd4323bca35f27a6a06cee 2023-04-13 20:55:39,343 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e5b993e8a7eb4efa8cb9588bfcf62706 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/e5b993e8a7eb4efa8cb9588bfcf62706 2023-04-13 20:55:39,344 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/6ff682b45355440ba98cdb4f04242903 2023-04-13 20:55:39,345 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/a3fe0ac8ab874a828df1292a7e60098b 2023-04-13 20:55:39,346 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/78fc10e1d0854d83823404f2cacff3ca to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/78fc10e1d0854d83823404f2cacff3ca 2023-04-13 20:55:39,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/fbb82b3cc466494aad0bbbcc73e27041 2023-04-13 20:55:39,347 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8e849eb1a19547b093d35e582a6d5536 2023-04-13 20:55:39,348 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/c88da419f6b3403eac238d7c17d89780 2023-04-13 20:55:39,349 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/518dd7563b5742d9b7a1ce6181f9bd57 2023-04-13 20:55:39,350 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5d93d29438b34d02bf61417ca8ef0c7f 2023-04-13 20:55:39,351 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/3debd07b4fe444a6a4ca6ad1cdcc985b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/3debd07b4fe444a6a4ca6ad1cdcc985b 2023-04-13 20:55:39,352 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/38cf3dc928fb4a7fba3061f4c525a352 2023-04-13 20:55:39,353 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/dffe25fdf1c24cb6915a4c2663b9e1bf 2023-04-13 20:55:39,354 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/ad23118ef9aa49719e4dde9840cd2966 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/ad23118ef9aa49719e4dde9840cd2966 2023-04-13 20:55:39,356 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/af920f80c1884613985988dfdeee16d7 2023-04-13 20:55:39,357 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/2968a13292404c48b0706ad805258944 2023-04-13 20:55:39,358 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/5c5a8deb60a74568a5451e7e35b8cd4c 2023-04-13 20:55:39,363 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9ca124a56fd44a89a21f3aa590b812da, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e2227fc57f74697973dd4072ac621b3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/69fc854ae3884a2488937978ccdbbfb8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/d4d6733793e444fcb4320d3a1c89db59, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f1d937c4bd214f119eea7bb6dafb07b0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/62bb14dd07f24b8f904834a56a09397e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8] to archive 2023-04-13 20:55:39,367 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:39,369 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b1a0aafcce3347f1921ac6cd6f0afa9b 2023-04-13 20:55:39,370 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e72455e97a240bfaf5ff453665b665e 2023-04-13 20:55:39,371 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6ff7b76beb7b46b7a84485b3609f7f4c 2023-04-13 20:55:39,372 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9ca124a56fd44a89a21f3aa590b812da to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9ca124a56fd44a89a21f3aa590b812da 2023-04-13 20:55:39,373 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/81eaec7f24014cf2b831e00fa186f216 2023-04-13 20:55:39,374 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/ef52b306df4545eca526584fc9be8473 2023-04-13 20:55:39,375 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e2227fc57f74697973dd4072ac621b3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/0e2227fc57f74697973dd4072ac621b3 2023-04-13 20:55:39,376 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/71e79e9671994b87aaca87f0b5d91292 2023-04-13 20:55:39,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/cf04e3775d87467598bcad76a5859593 2023-04-13 20:55:39,377 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/69fc854ae3884a2488937978ccdbbfb8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/69fc854ae3884a2488937978ccdbbfb8 2023-04-13 20:55:39,378 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/5c8c06a59205488f9ac0996522757f1e 2023-04-13 20:55:39,379 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/a5a8567cf6cf431e83c5e151d51bf980 2023-04-13 20:55:39,380 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/d4d6733793e444fcb4320d3a1c89db59 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/d4d6733793e444fcb4320d3a1c89db59 2023-04-13 20:55:39,381 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/8734be5a9b23409e9f384e4837db27d0 2023-04-13 20:55:39,382 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/6fa4a2e7202f458a8ecfabbc3ac22fe1 2023-04-13 20:55:39,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/39b612a1db0e486296fd5349867805f9 2023-04-13 20:55:39,383 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/2cc781807b774a918ebed8bb8a8f47f7 2023-04-13 20:55:39,385 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b8a0dfbc10fb44db98153a16fafc430e 2023-04-13 20:55:39,386 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f1d937c4bd214f119eea7bb6dafb07b0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f1d937c4bd214f119eea7bb6dafb07b0 2023-04-13 20:55:39,386 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/792e561296a04b3c84c3cf55d595036f 2023-04-13 20:55:39,387 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/b0f7046ee8c4459a8f406d050c889223 2023-04-13 20:55:39,388 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/62bb14dd07f24b8f904834a56a09397e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/62bb14dd07f24b8f904834a56a09397e 2023-04-13 20:55:39,389 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/de80f91aae4d4d6da5d6222647df2738 2023-04-13 20:55:39,390 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/9647e901b1714397bf026310978c1f60 2023-04-13 20:55:39,391 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/fc50229c0f354817ba06af5611963be8 2023-04-13 20:55:39,562 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d420db70458e4732ae4494709e7e13de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d420db70458e4732ae4494709e7e13de 2023-04-13 20:55:39,967 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9deddbcb82924b9697921c28040b827c 2023-04-13 20:55:40,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/b05e00dd2a784c879b9303fe88c5ffe6 2023-04-13 20:55:40,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/f3df83acfdf84836b67802b038f88b79 2023-04-13 20:55:40,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9deddbcb82924b9697921c28040b827c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c 2023-04-13 20:55:40,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c, entries=150, sequenceid=451, filesize=25.4 K 2023-04-13 20:55:40,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/b05e00dd2a784c879b9303fe88c5ffe6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6 2023-04-13 20:55:40,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6, entries=150, sequenceid=451, filesize=11.9 K 2023-04-13 20:55:40,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/f3df83acfdf84836b67802b038f88b79 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79 2023-04-13 20:55:40,404 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79, entries=150, sequenceid=451, filesize=11.9 K 2023-04-13 20:55:40,404 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1254ms, sequenceid=451, compaction requested=true 2023-04-13 20:55:40,405 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:40,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:40,405 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:40,405 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:40,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:40,405 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:40,406 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:40,406 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/A is initiating minor compaction (all files) 2023-04-13 20:55:40,406 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/A in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:40,406 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d55c192d7d044a82a01c2ea782dfacc3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=77.1 K 2023-04-13 20:55:40,406 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:40,406 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/B is initiating minor compaction (all files) 2023-04-13 20:55:40,406 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/B in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:40,406 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/826d1988c4b3454d803be5f9117cc012, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.5 K 2023-04-13 20:55:40,407 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d55c192d7d044a82a01c2ea782dfacc3, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:40,407 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 826d1988c4b3454d803be5f9117cc012, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:40,407 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9740e6bc74654e03a1075f8ed3e970a1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1681419335807 2023-04-13 20:55:40,408 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8fad9ca2c381421bad0e238477f906ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1681419335807 2023-04-13 20:55:40,408 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9deddbcb82924b9697921c28040b827c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681419337010 2023-04-13 20:55:40,408 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b05e00dd2a784c879b9303fe88c5ffe6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681419337010 2023-04-13 20:55:40,415 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#B#compaction#430 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:40,417 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#A#compaction#431 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:40,430 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/48c546cdd6b04a4185aef1718cc9af17 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/48c546cdd6b04a4185aef1718cc9af17 2023-04-13 20:55:40,434 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/B of 53d52adae37ccaf3c7691f3ca2bf72d5 into 48c546cdd6b04a4185aef1718cc9af17(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:40,434 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:40,434 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/B, priority=13, startTime=1681419340405; duration=0sec 2023-04-13 20:55:40,434 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:40,435 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:40,436 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:40,436 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): 53d52adae37ccaf3c7691f3ca2bf72d5/C is initiating minor compaction (all files) 2023-04-13 20:55:40,436 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 53d52adae37ccaf3c7691f3ca2bf72d5/C in TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:40,436 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/bdf8121477d64927a2bbdd8d91a01278, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp, totalSize=36.5 K 2023-04-13 20:55:40,438 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bdf8121477d64927a2bbdd8d91a01278, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1681419335149 2023-04-13 20:55:40,438 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f2bcefe635ad44b4b2ec3040d26a95f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1681419335807 2023-04-13 20:55:40,439 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f3df83acfdf84836b67802b038f88b79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1681419337010 2023-04-13 20:55:40,440 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9453092db7f8466a8b8ed741192e703f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9453092db7f8466a8b8ed741192e703f 2023-04-13 20:55:40,445 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/A of 53d52adae37ccaf3c7691f3ca2bf72d5 into 9453092db7f8466a8b8ed741192e703f(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:40,445 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:40,445 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/A, priority=13, startTime=1681419340405; duration=0sec 2023-04-13 20:55:40,446 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:40,446 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): 53d52adae37ccaf3c7691f3ca2bf72d5#C#compaction#432 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:40,456 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/1e1e1aa3679d40dba31b01d9b2822808 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/1e1e1aa3679d40dba31b01d9b2822808 2023-04-13 20:55:40,460 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 53d52adae37ccaf3c7691f3ca2bf72d5/C of 53d52adae37ccaf3c7691f3ca2bf72d5 into 1e1e1aa3679d40dba31b01d9b2822808(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:40,460 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:40,460 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5., storeName=53d52adae37ccaf3c7691f3ca2bf72d5/C, priority=13, startTime=1681419340405; duration=0sec 2023-04-13 20:55:40,460 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:45,638 DEBUG [Thread-2644] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:54584 2023-04-13 20:55:45,639 DEBUG [Thread-2644] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:45,643 DEBUG [Thread-2646] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:54584 2023-04-13 20:55:45,643 DEBUG [Thread-2646] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:45,691 DEBUG [Thread-2638] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:54584 2023-04-13 20:55:45,691 DEBUG [Thread-2638] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:45,691 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:55:45,691 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 71 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 43 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1855 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5565 rows 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1834 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5498 rows 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1846 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5532 rows 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-04-13 20:55:45,692 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5603 rows 2023-04-13 20:55:45,693 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1842 2023-04-13 20:55:45,693 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5524 rows 2023-04-13 20:55:45,693 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:55:45,693 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:54584 2023-04-13 20:55:45,693 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:55:45,696 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:55:45,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:55:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:45,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:45,702 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419345702"}]},"ts":"1681419345702"} 2023-04-13 20:55:45,703 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:55:45,704 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:55:45,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, UNASSIGN}] 2023-04-13 20:55:45,706 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, UNASSIGN 2023-04-13 20:55:45,707 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:45,707 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419345707"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419345707"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419345707"}]},"ts":"1681419345707"} 2023-04-13 20:55:45,707 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:45,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:45,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 53d52adae37ccaf3c7691f3ca2bf72d5, disabling compactions & flushes 2023-04-13 20:55:45,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:45,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:45,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. after waiting 0 ms 2023-04-13 20:55:45,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:45,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 53d52adae37ccaf3c7691f3ca2bf72d5 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=A 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=B 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 53d52adae37ccaf3c7691f3ca2bf72d5, store=C 2023-04-13 20:55:45,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:46,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:46,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041319f8393818fc4405823f2e5983900143 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f8393818fc4405823f2e5983900143 2023-04-13 20:55:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:46,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=460, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9a7afd1fbefc4fcb9423b95f54730dd0 2023-04-13 20:55:46,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:47,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/372a55f3f78c436086e0b034645e3623 2023-04-13 20:55:47,492 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/c492a6a74158419196e7213196a6ffc4 2023-04-13 20:55:47,495 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/A/9a7afd1fbefc4fcb9423b95f54730dd0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9a7afd1fbefc4fcb9423b95f54730dd0 2023-04-13 20:55:47,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9a7afd1fbefc4fcb9423b95f54730dd0, entries=100, sequenceid=460, filesize=18.6 K 2023-04-13 20:55:47,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/B/372a55f3f78c436086e0b034645e3623 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/372a55f3f78c436086e0b034645e3623 2023-04-13 20:55:47,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/372a55f3f78c436086e0b034645e3623, entries=100, sequenceid=460, filesize=9.5 K 2023-04-13 20:55:47,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/.tmp/C/c492a6a74158419196e7213196a6ffc4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/c492a6a74158419196e7213196a6ffc4 2023-04-13 20:55:47,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/c492a6a74158419196e7213196a6ffc4, entries=100, sequenceid=460, filesize=9.5 K 2023-04-13 20:55:47,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 53d52adae37ccaf3c7691f3ca2bf72d5 in 1647ms, sequenceid=460, compaction requested=false 2023-04-13 20:55:47,509 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d55c192d7d044a82a01c2ea782dfacc3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c] to archive 2023-04-13 20:55:47,510 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:47,511 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d55c192d7d044a82a01c2ea782dfacc3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/d55c192d7d044a82a01c2ea782dfacc3 2023-04-13 20:55:47,512 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9740e6bc74654e03a1075f8ed3e970a1 2023-04-13 20:55:47,513 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9deddbcb82924b9697921c28040b827c 2023-04-13 20:55:47,517 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/826d1988c4b3454d803be5f9117cc012, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6] to archive 2023-04-13 20:55:47,518 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:47,519 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/826d1988c4b3454d803be5f9117cc012 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/826d1988c4b3454d803be5f9117cc012 2023-04-13 20:55:47,520 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/8fad9ca2c381421bad0e238477f906ee 2023-04-13 20:55:47,521 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/b05e00dd2a784c879b9303fe88c5ffe6 2023-04-13 20:55:47,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/bdf8121477d64927a2bbdd8d91a01278, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79] to archive 2023-04-13 20:55:47,525 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:55:47,526 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/bdf8121477d64927a2bbdd8d91a01278 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/bdf8121477d64927a2bbdd8d91a01278 2023-04-13 20:55:47,527 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f2bcefe635ad44b4b2ec3040d26a95f3 2023-04-13 20:55:47,528 DEBUG [StoreCloser-TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/f3df83acfdf84836b67802b038f88b79 2023-04-13 20:55:47,533 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits/463.seqid, newMaxSeqId=463, maxSeqId=4 2023-04-13 20:55:47,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5. 2023-04-13 20:55:47,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 53d52adae37ccaf3c7691f3ca2bf72d5: 2023-04-13 20:55:47,535 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=53d52adae37ccaf3c7691f3ca2bf72d5, regionState=CLOSED 2023-04-13 20:55:47,536 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419347535"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419347535"}]},"ts":"1681419347535"} 2023-04-13 20:55:47,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:47,538 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-13 20:55:47,538 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 53d52adae37ccaf3c7691f3ca2bf72d5, server=jenkins-hbase4.apache.org,36431,1681419217857 in 1.8290 sec 2023-04-13 20:55:47,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-13 20:55:47,539 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=53d52adae37ccaf3c7691f3ca2bf72d5, UNASSIGN in 1.8330 sec 2023-04-13 20:55:47,539 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419347539"}]},"ts":"1681419347539"} 2023-04-13 20:55:47,540 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:55:47,542 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:55:47,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8450 sec 2023-04-13 20:55:47,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-13 20:55:47,806 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-13 20:55:47,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:55:47,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,809 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 20:55:47,810 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,814 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:47,816 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits] 2023-04-13 20:55:47,818 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9453092db7f8466a8b8ed741192e703f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9453092db7f8466a8b8ed741192e703f 2023-04-13 20:55:47,820 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9a7afd1fbefc4fcb9423b95f54730dd0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/A/9a7afd1fbefc4fcb9423b95f54730dd0 2023-04-13 20:55:47,822 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/372a55f3f78c436086e0b034645e3623 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/372a55f3f78c436086e0b034645e3623 2023-04-13 20:55:47,824 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/48c546cdd6b04a4185aef1718cc9af17 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/B/48c546cdd6b04a4185aef1718cc9af17 2023-04-13 20:55:47,826 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/1e1e1aa3679d40dba31b01d9b2822808 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/1e1e1aa3679d40dba31b01d9b2822808 2023-04-13 20:55:47,827 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/c492a6a74158419196e7213196a6ffc4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/C/c492a6a74158419196e7213196a6ffc4 2023-04-13 20:55:47,829 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits/463.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5/recovered.edits/463.seqid 2023-04-13 20:55:47,830 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/53d52adae37ccaf3c7691f3ca2bf72d5 2023-04-13 20:55:47,830 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:55:47,830 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:55:47,831 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 20:55:47,834 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313bf2466df614dbfb3eb34908fc4a563 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041313bf2466df614dbfb3eb34908fc4a563 2023-04-13 20:55:47,835 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413160e8965e9d8424799aa10cf1cac3795 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413160e8965e9d8424799aa10cf1cac3795 2023-04-13 20:55:47,836 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413191435be04da4a229aa16ef6a5c80ee2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413191435be04da4a229aa16ef6a5c80ee2 2023-04-13 20:55:47,837 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f8393818fc4405823f2e5983900143 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041319f8393818fc4405823f2e5983900143 2023-04-13 20:55:47,838 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132337d3c333134048b15f28e478ce60ea to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132337d3c333134048b15f28e478ce60ea 2023-04-13 20:55:47,839 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132d6b6c6a228849d88bae969848394cba to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132d6b6c6a228849d88bae969848394cba 2023-04-13 20:55:47,840 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134200b3ffae2f44ea808cc6acb8289898 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134200b3ffae2f44ea808cc6acb8289898 2023-04-13 20:55:47,841 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041343d6f92a13e243a9a2e67ae4cf68fd10 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041343d6f92a13e243a9a2e67ae4cf68fd10 2023-04-13 20:55:47,842 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134d57b81343d9439ab5858b12af32d643 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304134d57b81343d9439ab5858b12af32d643 2023-04-13 20:55:47,843 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135fd5e5988172463abde9f2e5ac647256 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135fd5e5988172463abde9f2e5ac647256 2023-04-13 20:55:47,844 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413684704d3dfee4b80926411114c30c087 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413684704d3dfee4b80926411114c30c087 2023-04-13 20:55:47,845 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041374efd44f00314a72a16699790e99d783 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041374efd44f00314a72a16699790e99d783 2023-04-13 20:55:47,846 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041378ac8c9c02834a9c8a62b12d9c910349 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041378ac8c9c02834a9c8a62b12d9c910349 2023-04-13 20:55:47,847 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137c9541f26c6d49b58a4dbb7a1de4b8b4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137c9541f26c6d49b58a4dbb7a1de4b8b4 2023-04-13 20:55:47,848 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385e0d3bff7244f75afc6767f1ddeccdc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041385e0d3bff7244f75afc6767f1ddeccdc 2023-04-13 20:55:47,848 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e69dd1987654c089ccc490747bfc8b5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138e69dd1987654c089ccc490747bfc8b5 2023-04-13 20:55:47,849 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ae7b753831674536955811333d3e8f46 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413ae7b753831674536955811333d3e8f46 2023-04-13 20:55:47,850 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b0495da185b0406babb6deeb87b8641f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413b0495da185b0406babb6deeb87b8641f 2023-04-13 20:55:47,851 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cd6d54aadde54b8abb656aa6ce9ab55f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cd6d54aadde54b8abb656aa6ce9ab55f 2023-04-13 20:55:47,852 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3c59177f38244ec8ee7f0d761057ffe to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d3c59177f38244ec8ee7f0d761057ffe 2023-04-13 20:55:47,853 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d420db70458e4732ae4494709e7e13de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d420db70458e4732ae4494709e7e13de 2023-04-13 20:55:47,853 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413da80283b715c4c00935c4890c626c517 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413da80283b715c4c00935c4890c626c517 2023-04-13 20:55:47,854 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:55:47,856 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,857 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:55:47,858 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:55:47,859 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,859 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:55:47,859 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419347859"}]},"ts":"9223372036854775807"} 2023-04-13 20:55:47,860 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:55:47,860 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 53d52adae37ccaf3c7691f3ca2bf72d5, NAME => 'TestAcidGuarantees,,1681419314024.53d52adae37ccaf3c7691f3ca2bf72d5.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:55:47,860 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:55:47,860 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419347860"}]},"ts":"9223372036854775807"} 2023-04-13 20:55:47,861 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:55:47,863 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,864 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 56 msec 2023-04-13 20:55:47,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-13 20:55:47,911 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-13 20:55:47,922 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 228), OpenFileDescriptor=526 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=369 (was 336) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=735 (was 582) - AvailableMemoryMB LEAK? - 2023-04-13 20:55:47,933 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=526, MaxFileDescriptor=60000, SystemLoadAverage=369, ProcessCount=173, AvailableMemoryMB=735 2023-04-13 20:55:47,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:55:47,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:55:47,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:55:47,937 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:55:47,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-13 20:55:47,937 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:55:47,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 20:55:47,938 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:47,939 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 empty. 2023-04-13 20:55:47,939 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:47,939 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 20:55:48,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 20:55:48,346 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:55:48,348 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b4116c26d7fbdc1e912f851d9767a8d2, NAME => 'TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:55:48,353 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:55:48,353 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b4116c26d7fbdc1e912f851d9767a8d2, disabling compactions & flushes 2023-04-13 20:55:48,354 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,354 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,354 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. after waiting 0 ms 2023-04-13 20:55:48,354 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,354 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,354 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:48,356 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:55:48,356 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419348356"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419348356"}]},"ts":"1681419348356"} 2023-04-13 20:55:48,357 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:55:48,358 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:55:48,358 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419348358"}]},"ts":"1681419348358"} 2023-04-13 20:55:48,359 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:55:48,362 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, ASSIGN}] 2023-04-13 20:55:48,364 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, ASSIGN 2023-04-13 20:55:48,364 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:55:48,516 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=b4116c26d7fbdc1e912f851d9767a8d2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:48,516 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419348515"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419348515"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419348515"}]},"ts":"1681419348515"} 2023-04-13 20:55:48,517 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:55:48,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 20:55:48,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b4116c26d7fbdc1e912f851d9767a8d2, NAME => 'TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:55:48,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:55:48,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,673 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,674 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A 2023-04-13 20:55:48,674 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A 2023-04-13 20:55:48,674 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:48,674 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b4116c26d7fbdc1e912f851d9767a8d2 columnFamilyName A 2023-04-13 20:55:48,675 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(310): Store=b4116c26d7fbdc1e912f851d9767a8d2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:48,675 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,676 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B 2023-04-13 20:55:48,676 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B 2023-04-13 20:55:48,676 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:48,676 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b4116c26d7fbdc1e912f851d9767a8d2 columnFamilyName B 2023-04-13 20:55:48,677 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(310): Store=b4116c26d7fbdc1e912f851d9767a8d2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:48,677 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,678 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C 2023-04-13 20:55:48,678 DEBUG [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C 2023-04-13 20:55:48,678 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:55:48,678 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b4116c26d7fbdc1e912f851d9767a8d2 columnFamilyName C 2023-04-13 20:55:48,679 INFO [StoreOpener-b4116c26d7fbdc1e912f851d9767a8d2-1] regionserver.HStore(310): Store=b4116c26d7fbdc1e912f851d9767a8d2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:55:48,679 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:55:48,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:48,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:55:48,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b4116c26d7fbdc1e912f851d9767a8d2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75202863, jitterRate=0.12060998380184174}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:55:48,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:48,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., pid=49, masterSystemTime=1681419348668 2023-04-13 20:55:48,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,685 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:48,685 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=b4116c26d7fbdc1e912f851d9767a8d2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:48,685 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419348685"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419348685"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419348685"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419348685"}]},"ts":"1681419348685"} 2023-04-13 20:55:48,687 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-13 20:55:48,687 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 in 169 msec 2023-04-13 20:55:48,688 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-13 20:55:48,688 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, ASSIGN in 325 msec 2023-04-13 20:55:48,689 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:55:48,689 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419348689"}]},"ts":"1681419348689"} 2023-04-13 20:55:48,690 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:55:48,692 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:55:48,692 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 757 msec 2023-04-13 20:55:49,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-13 20:55:49,041 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-13 20:55:49,042 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x37705401 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,046 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ec943a9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,047 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,050 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,050 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:55:49,052 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:55:49,053 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x15880ffb to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,058 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20ed9117, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,059 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x052203cb to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,062 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50002343, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,063 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x420f945a to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,067 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5726fb8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,067 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x51b8df2a to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,070 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f2b3c03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,071 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,078 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,078 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,082 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,082 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,087 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,088 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x659c68ab to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,091 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@549bef45, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,091 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x680e773e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,096 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ed183a4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,096 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x0b8f652e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:55:49,103 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10a344e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:55:49,109 DEBUG [hconnection-0x1be275f1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,111 DEBUG [hconnection-0x182c66f5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,112 DEBUG [hconnection-0x5dfb1d8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,113 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40072, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:49,116 DEBUG [hconnection-0x6dedc2c0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,121 DEBUG [hconnection-0x7b067159-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,121 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,121 DEBUG [hconnection-0x6fd6413d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,122 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40112, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,126 DEBUG [hconnection-0x7ef6a774-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,128 DEBUG [hconnection-0x514fed9c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,128 DEBUG [hconnection-0x4489e751-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,129 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40120, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,130 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40060, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,130 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,132 DEBUG [hconnection-0x51f3f968-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:55:49,134 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40134, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,134 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,134 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,137 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:55:49,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:49,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:49,145 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:49,145 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:49,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:49,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:49,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:49,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,147 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,148 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:49,152 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:49,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:49,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:49,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:49,155 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:49,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/b3aeab3fb73a46c884d5814194fad692 2023-04-13 20:55:49,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca267c3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:49,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:49,165 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:49,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:49,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:49,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:49,170 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:49,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:49,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:49,170 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:49,171 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:49,173 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,173 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,173 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:49,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,188 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:49,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,188 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:49,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419409165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419409171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1681419349163, max timestamp=1681419349175Num uniques 859; 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9554, totalHeapSize=21976, min timestamp=1681419349163, max timestamp=1681419349175Num uniques 859; 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=9554, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419349163, max timestamp=1681419349175Num uniques 859; 2023-04-13 20:55:49,197 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39324, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-13 20:55:49,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419409219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419409224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419409226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,288 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:49,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:49,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:49,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419409298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419409298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:49,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:49,300 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,302 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:49,302 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:49,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:49,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:49,304 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,304 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:49,304 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:49,304 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:49,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:49,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:49,309 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,311 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,311 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:49,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f77d4d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:49,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:49,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,312 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:49,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:49,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,315 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:49,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:49,317 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:49,317 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:49,317 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:49,317 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:49,317 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:49,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,323 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,324 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:49,324 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:49,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:49,323 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,324 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419409327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419409327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419409330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,424 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:49,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:49,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:49,429 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:49,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:49,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:49,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:49,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:49,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:49,438 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:49,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:49,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202f89d7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:49,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:49,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,443 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:49,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:49,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:49,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:49,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:49,447 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:49,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:49,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:49,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:49,447 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:49,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,461 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,462 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,463 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,471 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:49,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:49,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,472 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:49,472 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,472 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419409500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419409501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419409528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419409530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419409532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,572 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:49,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:49,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:49,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:49,575 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:49,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:49,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:49,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:49,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:49,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:49,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:49,580 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:49,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f7feba5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:49,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:49,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:49,583 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:49,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:49,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:49,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,586 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/9da682cd17f4447490eb89ec730b9781 2023-04-13 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:49,588 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:49,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:49,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:49,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:49,588 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:49,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,590 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,590 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:49,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:49,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:49,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:49,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:49,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:49,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:49,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:49,597 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:49,597 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:49,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/782ae9c9a253406f8e568586222d23cd 2023-04-13 20:55:49,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/b3aeab3fb73a46c884d5814194fad692 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692 2023-04-13 20:55:49,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692, entries=150, sequenceid=12, filesize=11.6 K 2023-04-13 20:55:49,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/9da682cd17f4447490eb89ec730b9781 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781 2023-04-13 20:55:49,623 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781, entries=150, sequenceid=12, filesize=11.6 K 2023-04-13 20:55:49,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/782ae9c9a253406f8e568586222d23cd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd 2023-04-13 20:55:49,627 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd, entries=150, sequenceid=12, filesize=11.6 K 2023-04-13 20:55:49,628 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=124.85 KB/127846 for b4116c26d7fbdc1e912f851d9767a8d2 in 483ms, sequenceid=12, compaction requested=false 2023-04-13 20:55:49,628 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:49,697 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:49,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:49,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:49,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:49,701 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,701 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:49,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:49,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:49,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:49,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:49,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:49,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:49,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:49,708 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=124.85 KB heapSize=354.38 KB 2023-04-13 20:55:49,708 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:49,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:49,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:49,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:49,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:49,709 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:49,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:49,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419409824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419409830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419409837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419409838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419409843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419409933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:49,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419409933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,116 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/bac28b76070b489fb9ea78a419aa2a2a 2023-04-13 20:55:50,128 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/6ec1046aa4674acf9fc8aca0d11a5a68 2023-04-13 20:55:50,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419410137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419410137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419410344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419410348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419410354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,445 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419410442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419410442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,537 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=39.87 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0d352663878e48b2977a1abedef418c5 2023-04-13 20:55:50,541 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/bac28b76070b489fb9ea78a419aa2a2a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a 2023-04-13 20:55:50,544 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 20:55:50,545 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/6ec1046aa4674acf9fc8aca0d11a5a68 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68 2023-04-13 20:55:50,549 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 20:55:50,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0d352663878e48b2977a1abedef418c5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5 2023-04-13 20:55:50,553 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5, entries=150, sequenceid=47, filesize=11.6 K 2023-04-13 20:55:50,553 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~124.85 KB/127846, heapSize ~354.34 KB/362840, currentSize=53.67 KB/54960 for b4116c26d7fbdc1e912f851d9767a8d2 in 845ms, sequenceid=47, compaction requested=false 2023-04-13 20:55:50,554 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:50,554 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:50,554 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:50,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@304c1bf6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:50,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:50,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,559 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:50,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:50,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:50,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:50,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:50,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:50,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:50,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:50,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:50,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:50,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:50,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:50,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:50,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:50,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:50,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:50,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:50,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,572 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:50,573 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:50,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:50,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:50,673 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:50,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:50,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:50,679 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:50,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:50,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:50,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:50,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:50,680 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,680 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:50,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:50,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:50,681 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:50,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:50,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:50,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:50,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:50,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:50,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:50,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:50,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:50,682 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:50,683 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:50,683 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:50,683 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:50,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:50,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681419350970, max timestamp=1681419350995Num uniques 859; 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681419350970, max timestamp=1681419350995Num uniques 859; 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1681419350970, max timestamp=1681419350995Num uniques 859; 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 20:55:50,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 20:55:51,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,041 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419411039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,092 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/67c072c553864608824365bfc579bcc0 2023-04-13 20:55:51,103 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/25ae632af91e4daa905fe04e1e8cb673 2023-04-13 20:55:51,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419411142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419411347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419411353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419411360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419411367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,513 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/2881a18233514adc98364c6b3d60b2cc 2023-04-13 20:55:51,517 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/67c072c553864608824365bfc579bcc0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0 2023-04-13 20:55:51,521 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0, entries=150, sequenceid=58, filesize=11.6 K 2023-04-13 20:55:51,521 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/25ae632af91e4daa905fe04e1e8cb673 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673 2023-04-13 20:55:51,529 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673, entries=150, sequenceid=58, filesize=11.6 K 2023-04-13 20:55:51,530 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/2881a18233514adc98364c6b3d60b2cc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc 2023-04-13 20:55:51,534 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc, entries=150, sequenceid=58, filesize=11.6 K 2023-04-13 20:55:51,535 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for b4116c26d7fbdc1e912f851d9767a8d2 in 852ms, sequenceid=58, compaction requested=true 2023-04-13 20:55:51,535 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:51,535 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:51,535 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:51,536 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:51,537 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,539 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:51,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ced15a7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:51,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:51,540 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,542 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:51,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:51,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,542 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:51,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:51,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:51,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:51,546 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:51,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:51,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:51,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:51,546 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:51,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:51,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,550 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:51,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:51,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:51,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:51,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:51,552 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:51,552 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:51,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:51,652 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:51,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:51,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:51,655 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:51,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:51,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,659 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:51,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:51,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:51,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:51,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:51,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:51,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:51,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:51,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:51,661 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:51,661 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-13 20:55:51,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:51,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:51,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419411688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:51,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:51,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419411793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419411998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419412000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,069 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ffd172ee39d04ba3b1463d12db0eb3c4 2023-04-13 20:55:52,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419412311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419412312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,480 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/12637c40d96940ef86ead80b09ead397 2023-04-13 20:55:52,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419412826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:52,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419412826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,891 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=93 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/38486c5ebbbf401f845c98d80a310833 2023-04-13 20:55:52,895 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ffd172ee39d04ba3b1463d12db0eb3c4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4 2023-04-13 20:55:52,899 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4, entries=200, sequenceid=93, filesize=13.9 K 2023-04-13 20:55:52,899 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/12637c40d96940ef86ead80b09ead397 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397 2023-04-13 20:55:52,902 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397, entries=150, sequenceid=93, filesize=11.6 K 2023-04-13 20:55:52,903 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/38486c5ebbbf401f845c98d80a310833 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833 2023-04-13 20:55:52,906 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833, entries=150, sequenceid=93, filesize=11.6 K 2023-04-13 20:55:52,907 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 1246ms, sequenceid=93, compaction requested=true 2023-04-13 20:55:52,907 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:52,907 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:52,910 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,910 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,911 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:52,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d054dc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:52,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:52,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:52,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:52,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:52,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:52,918 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:52,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:52,921 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:52,921 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:52,921 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:52,921 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:52,921 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:52,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,923 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:52,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:52,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:52,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,926 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:52,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:52,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:52,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:52,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:52,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:52,928 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:52,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,928 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:52,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:52,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:52,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:53,029 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:53,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:53,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:53,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:53,032 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:53,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:53,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:53,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:53,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:53,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:53,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:53,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:53,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:53,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:53,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:53,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:53,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:53,035 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:53,035 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:53,035 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:53,035 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:53,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:53,062 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:55:53,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:53,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:53,446 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/582f5d012ae5438dbac55f8830776994 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419353418, max timestamp=1681419353448Num uniques 859; 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419353418, max timestamp=1681419353448Num uniques 859; 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419353418, max timestamp=1681419353448Num uniques 859; 2023-04-13 20:55:53,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:53,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419413485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419413491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419413492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419413593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419413600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419413607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419413805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419413812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419413816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419413830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:53,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419413835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,858 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/1145ad3bc154436d8962aafc51bac2f4 2023-04-13 20:55:53,867 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c6834b64b533462e8bbd42af57b4e44b 2023-04-13 20:55:53,871 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/582f5d012ae5438dbac55f8830776994 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994 2023-04-13 20:55:53,876 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994, entries=150, sequenceid=103, filesize=11.6 K 2023-04-13 20:55:53,876 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/1145ad3bc154436d8962aafc51bac2f4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4 2023-04-13 20:55:53,880 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4, entries=150, sequenceid=103, filesize=11.6 K 2023-04-13 20:55:53,881 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c6834b64b533462e8bbd42af57b4e44b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b 2023-04-13 20:55:53,884 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b, entries=150, sequenceid=103, filesize=11.6 K 2023-04-13 20:55:53,884 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b4116c26d7fbdc1e912f851d9767a8d2 in 849ms, sequenceid=103, compaction requested=true 2023-04-13 20:55:53,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:53,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:53,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:53,890 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:53,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:53,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3092afa0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:53,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:53,891 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,892 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:53,893 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:53,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,893 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:53,894 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:53,894 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:53,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:53,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:53,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:53,897 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:53,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:53,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:53,897 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:53,897 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:53,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:53,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,898 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:53,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,900 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:53,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:53,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:53,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:53,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:53,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:53,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:53,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:53,904 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:53,904 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:53,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:53,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:53,904 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:54,004 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:54,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:54,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:54,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:54,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:54,007 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:54,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:54,007 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:54,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,009 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,009 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:54,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:54,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:54,011 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:54,011 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,011 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:54,011 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:54,011 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:54,011 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,011 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:54,011 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-13 20:55:54,011 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:54,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,019 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ece2fd786c2e48289592d2483e1ce480 2023-04-13 20:55:54,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:54,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:54,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419414134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419414135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419414136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419414238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419414244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419414250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,431 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/037b7e21b9c1465b966061331f61d29c 2023-04-13 20:55:54,442 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=138 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/14d0d3ea2f06468990e51d04dafce613 2023-04-13 20:55:54,445 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ece2fd786c2e48289592d2483e1ce480 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480 2023-04-13 20:55:54,450 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480, entries=150, sequenceid=138, filesize=11.8 K 2023-04-13 20:55:54,450 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/037b7e21b9c1465b966061331f61d29c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c 2023-04-13 20:55:54,454 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c, entries=150, sequenceid=138, filesize=11.8 K 2023-04-13 20:55:54,455 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/14d0d3ea2f06468990e51d04dafce613 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613 2023-04-13 20:55:54,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419414450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419414451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,461 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613, entries=150, sequenceid=138, filesize=11.8 K 2023-04-13 20:55:54,461 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 450ms, sequenceid=138, compaction requested=true 2023-04-13 20:55:54,461 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:54,461 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,461 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:54,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:54,466 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:54,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605181ef[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:54,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:54,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:54,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:54,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:54,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:54,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/4fc2da1df245409b8c35c30e09e01afa 2023-04-13 20:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:54,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:54,474 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:54,475 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:54,475 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:54,475 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:54,475 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:54,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,483 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:54,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:54,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,484 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,484 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:54,484 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:54,484 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,584 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:54,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:54,586 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:54,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:54,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:54,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:54,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:54,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:54,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:54,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:54,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:54,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:54,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5f06f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:54,592 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:54,593 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,594 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:54,594 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:54,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:54,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,596 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:54,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:54,598 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:54,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:54,598 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:54,601 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,601 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,601 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,601 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:54,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,605 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:54,605 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,605 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419354622, max timestamp=1681419354660Num uniques 859; 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419354622, max timestamp=1681419354660Num uniques 859; 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419354622, max timestamp=1681419354660Num uniques 859; 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:54,662 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:54,705 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:54,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:54,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:54,708 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:54,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:54,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:54,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:54,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:54,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:54,713 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:54,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:54,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d13299[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:54,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:54,717 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:54,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:54,721 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:54,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:54,722 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,722 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:54,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:54,725 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:54,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:54,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:54,725 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:54,725 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:54,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,726 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,730 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:54,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,730 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,731 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:54,731 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:54,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419414761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419414768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419414769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,831 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:54,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:54,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:54,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:54,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:54,834 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:54,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:54,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:54,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:54,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:54,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:54,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:54,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:54,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:54,838 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:54,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:54,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@731b784c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:54,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:54,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,843 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:54,843 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:54,845 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:54,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:54,847 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:54,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:54,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:54,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:54,847 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:54,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:54,849 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:54,849 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,849 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,855 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:54,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:54,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:54,855 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:54,855 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:54,855 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:54,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419414872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419414872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c9b52998fee7466db3949901f9f0a7f6 2023-04-13 20:55:54,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419414878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/f625e1ca168b4fc097c786b2f558489b 2023-04-13 20:55:54,897 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/4fc2da1df245409b8c35c30e09e01afa as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa 2023-04-13 20:55:54,900 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa, entries=200, sequenceid=149, filesize=14.1 K 2023-04-13 20:55:54,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c9b52998fee7466db3949901f9f0a7f6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6 2023-04-13 20:55:54,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6, entries=150, sequenceid=149, filesize=11.8 K 2023-04-13 20:55:54,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/f625e1ca168b4fc097c786b2f558489b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b 2023-04-13 20:55:54,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b, entries=150, sequenceid=149, filesize=11.8 K 2023-04-13 20:55:54,916 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b4116c26d7fbdc1e912f851d9767a8d2 in 451ms, sequenceid=149, compaction requested=true 2023-04-13 20:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:54,916 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 20:55:54,916 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 20:55:54,916 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:54,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:54,919 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 88232 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 20:55:54,919 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:55:54,919 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,919 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=86.2 K 2023-04-13 20:55:54,919 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83502 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 20:55:54,919 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:55:54,919 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,920 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=81.5 K 2023-04-13 20:55:54,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b3aeab3fb73a46c884d5814194fad692, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419349139 2023-04-13 20:55:54,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9da682cd17f4447490eb89ec730b9781, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419349139 2023-04-13 20:55:54,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting bac28b76070b489fb9ea78a419aa2a2a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681419349218 2023-04-13 20:55:54,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6ec1046aa4674acf9fc8aca0d11a5a68, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681419349218 2023-04-13 20:55:54,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 67c072c553864608824365bfc579bcc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681419349811 2023-04-13 20:55:54,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 25ae632af91e4daa905fe04e1e8cb673, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681419349811 2023-04-13 20:55:54,923 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ffd172ee39d04ba3b1463d12db0eb3c4, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681419351033 2023-04-13 20:55:54,924 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 12637c40d96940ef86ead80b09ead397, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681419351033 2023-04-13 20:55:54,924 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 582f5d012ae5438dbac55f8830776994, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681419351672 2023-04-13 20:55:54,924 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1145ad3bc154436d8962aafc51bac2f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681419351672 2023-04-13 20:55:54,924 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ece2fd786c2e48289592d2483e1ce480, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681419353471 2023-04-13 20:55:54,925 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 037b7e21b9c1465b966061331f61d29c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681419353471 2023-04-13 20:55:54,925 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 4fc2da1df245409b8c35c30e09e01afa, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354133 2023-04-13 20:55:54,925 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c9b52998fee7466db3949901f9f0a7f6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354134 2023-04-13 20:55:54,939 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#457 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:54,943 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#458 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:54,956 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:54,957 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/f298fbcf80ee47ebaf14f991bf22b67f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/f298fbcf80ee47ebaf14f991bf22b67f 2023-04-13 20:55:54,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:54,970 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into f298fbcf80ee47ebaf14f991bf22b67f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:54,970 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:54,970 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:54,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,970 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=9, startTime=1681419354916; duration=0sec 2023-04-13 20:55:54,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:54,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:54,970 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:54,970 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-13 20:55:54,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,973 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:54,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:54,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,974 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:54,974 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:54,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:54,974 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83502 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-13 20:55:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:54,974 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:55:54,974 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:54,974 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,975 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=81.5 K 2023-04-13 20:55:54,975 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 782ae9c9a253406f8e568586222d23cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1681419349139 2023-04-13 20:55:54,976 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/e77c66fa8b1041ac9d409b17008e3a43 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e77c66fa8b1041ac9d409b17008e3a43 2023-04-13 20:55:54,976 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0d352663878e48b2977a1abedef418c5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1681419349218 2023-04-13 20:55:54,976 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2881a18233514adc98364c6b3d60b2cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681419349811 2023-04-13 20:55:54,977 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 38486c5ebbbf401f845c98d80a310833, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=93, earliestPutTs=1681419351033 2023-04-13 20:55:54,977 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c6834b64b533462e8bbd42af57b4e44b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1681419351672 2023-04-13 20:55:54,977 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 14d0d3ea2f06468990e51d04dafce613, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=138, earliestPutTs=1681419353471 2023-04-13 20:55:54,978 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f625e1ca168b4fc097c786b2f558489b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354134 2023-04-13 20:55:54,979 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into e77c66fa8b1041ac9d409b17008e3a43(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:54,979 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:54,979 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=9, startTime=1681419354916; duration=0sec 2023-04-13 20:55:54,980 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:54,981 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:54,981 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:54,982 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:54,982 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:54,982 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:54,982 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-13 20:55:54,982 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:54,982 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,983 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:54,983 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,983 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:54,983 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:54,995 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#460 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:54,995 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/143f1bdf011548708a8a6669244d661c 2023-04-13 20:55:55,011 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/775e93b595ba43caaeb35f8cb36bf5a6 2023-04-13 20:55:55,021 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=187 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/e96081a21d7c43aa8f1e247c1af6a4cf 2023-04-13 20:55:55,025 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/143f1bdf011548708a8a6669244d661c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c 2023-04-13 20:55:55,029 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 20:55:55,029 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/775e93b595ba43caaeb35f8cb36bf5a6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6 2023-04-13 20:55:55,032 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 20:55:55,033 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/e96081a21d7c43aa8f1e247c1af6a4cf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf 2023-04-13 20:55:55,036 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf, entries=150, sequenceid=187, filesize=11.8 K 2023-04-13 20:55:55,037 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for b4116c26d7fbdc1e912f851d9767a8d2 in 55ms, sequenceid=187, compaction requested=false 2023-04-13 20:55:55,037 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:55,037 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:55,037 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:55,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,038 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:55,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f29d139[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:55,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:55,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,041 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,041 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:55,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:55,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:55,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:55,053 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:55,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:55,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:55,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:55,053 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:55,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,056 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,060 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,062 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:55,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:55,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,063 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:55,063 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:55,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:55,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:55,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:55,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/fb9deae2c9db4bec9a17c3549f9f6e02 2023-04-13 20:55:55,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419355114, max timestamp=1681419355147Num uniques 859; 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419355114, max timestamp=1681419355147Num uniques 859; 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419355114, max timestamp=1681419355147Num uniques 859; 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:55,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419415147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:55,163 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:55,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:55,167 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:55,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:55,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:55,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,169 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:55,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:55,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:55,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:55,171 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,171 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:55,171 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:55,171 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:55,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:55,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:55,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:55,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34f42c3b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:55,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:55,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:55,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:55,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:55,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:55,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:55,185 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:55,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:55,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:55,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:55,185 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:55,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,188 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:55,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:55,191 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,191 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,192 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,192 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:55,192 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419415193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,210 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419415201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419415250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,292 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:55,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:55,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:55,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:55,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:55,297 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:55,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:55,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:55,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:55,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:55,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:55,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:55,302 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,302 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:55,302 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:55,302 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,303 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b6e85c6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:55,305 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:55,305 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:55,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:55,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:55,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419415302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,313 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:55,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:55,316 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:55,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:55,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:55,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:55,316 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:55,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419415311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:55,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:55,325 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,325 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:55,326 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:55,325 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,326 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,326 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,326 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,415 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/e1524e8f7f6d434db2c6879b78057733 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e1524e8f7f6d434db2c6879b78057733 2023-04-13 20:55:55,419 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into e1524e8f7f6d434db2c6879b78057733(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:55,419 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:55,419 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=9, startTime=1681419354917; duration=0sec 2023-04-13 20:55:55,419 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:55,426 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:55,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:55,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:55,430 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:55,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:55,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:55,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,431 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,434 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:55,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,435 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:55,438 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@431f49ff[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:55,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:55,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:55,441 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,441 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:55,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,443 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:55,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:55,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:55,447 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:55,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:55,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:55,447 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:55,447 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:55,449 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,451 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:55,454 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:55,456 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,456 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:55,456 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:55,456 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:55,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:55,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419415463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419415510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/bf5a65dde4cb46fcaa5bd886ac91ecc8 2023-04-13 20:55:55,526 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0e81e997450d491fbdd2712d0a056b40 2023-04-13 20:55:55,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/fb9deae2c9db4bec9a17c3549f9f6e02 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02 2023-04-13 20:55:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419415525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02, entries=200, sequenceid=197, filesize=14.1 K 2023-04-13 20:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/bf5a65dde4cb46fcaa5bd886ac91ecc8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8 2023-04-13 20:55:55,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8, entries=150, sequenceid=197, filesize=11.8 K 2023-04-13 20:55:55,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0e81e997450d491fbdd2712d0a056b40 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40 2023-04-13 20:55:55,539 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40, entries=150, sequenceid=197, filesize=11.8 K 2023-04-13 20:55:55,539 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b4116c26d7fbdc1e912f851d9767a8d2 in 442ms, sequenceid=197, compaction requested=true 2023-04-13 20:55:55,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:55,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:55,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:55,540 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:55,540 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:55,540 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:55,540 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:55,541 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38738 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:55,541 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:55:55,541 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:55:55,541 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,541 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,541 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e77c66fa8b1041ac9d409b17008e3a43, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=35.5 K 2023-04-13 20:55:55,541 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/f298fbcf80ee47ebaf14f991bf22b67f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=37.8 K 2023-04-13 20:55:55,541 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e77c66fa8b1041ac9d409b17008e3a43, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354134 2023-04-13 20:55:55,541 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f298fbcf80ee47ebaf14f991bf22b67f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354134 2023-04-13 20:55:55,542 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 775e93b595ba43caaeb35f8cb36bf5a6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681419354747 2023-04-13 20:55:55,543 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 143f1bdf011548708a8a6669244d661c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681419354747 2023-04-13 20:55:55,543 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bf5a65dde4cb46fcaa5bd886ac91ecc8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:55,543 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fb9deae2c9db4bec9a17c3549f9f6e02, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:55,548 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:55,549 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#466 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:55,556 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:55,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:55,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:55,560 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:55,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:55,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:55,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:55,563 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/8912cff09a3f4c6dbc5686f68df7b48b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8912cff09a3f4c6dbc5686f68df7b48b 2023-04-13 20:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:55,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:55,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:55,566 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/b06f48a4484a4702a041964be1cf48db as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b06f48a4484a4702a041964be1cf48db 2023-04-13 20:55:55,568 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:55,569 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:55,569 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-13 20:55:55,569 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 8912cff09a3f4c6dbc5686f68df7b48b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:55,569 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:55,569 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419355540; duration=0sec 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:55,569 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:55,569 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:55,569 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:55,570 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:55,570 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:55,570 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into b06f48a4484a4702a041964be1cf48db(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:55,570 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:55,570 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419355540; duration=0sec 2023-04-13 20:55:55,570 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:55,571 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:55,571 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:55:55,571 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:55,571 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e1524e8f7f6d434db2c6879b78057733, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=35.5 K 2023-04-13 20:55:55,572 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e1524e8f7f6d434db2c6879b78057733, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1681419354134 2023-04-13 20:55:55,572 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e96081a21d7c43aa8f1e247c1af6a4cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=187, earliestPutTs=1681419354747 2023-04-13 20:55:55,572 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0e81e997450d491fbdd2712d0a056b40, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:55,578 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:55,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:55,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:55,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419415816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419415824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419415836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419415842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,853 DEBUG [Thread-3428] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:55,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419415851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,861 DEBUG [Thread-3432] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:55:55,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1681419415936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:55,995 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e2ab2b2830344003bf198a6122848423 2023-04-13 20:55:56,000 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/02503561bbe9405e877f0fab1173cc17 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/02503561bbe9405e877f0fab1173cc17 2023-04-13 20:55:56,007 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 02503561bbe9405e877f0fab1173cc17(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:56,007 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:56,007 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=13, startTime=1681419355540; duration=0sec 2023-04-13 20:55:56,007 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:56,014 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/433e50d98a8e43e0a9f04e9adfe162bb 2023-04-13 20:55:56,024 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a1c9eb96b6c24bc09518ca38ae6001f0 2023-04-13 20:55:56,028 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e2ab2b2830344003bf198a6122848423 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423 2023-04-13 20:55:56,031 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423, entries=150, sequenceid=234, filesize=11.8 K 2023-04-13 20:55:56,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/433e50d98a8e43e0a9f04e9adfe162bb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb 2023-04-13 20:55:56,035 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb, entries=150, sequenceid=234, filesize=11.8 K 2023-04-13 20:55:56,036 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a1c9eb96b6c24bc09518ca38ae6001f0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0 2023-04-13 20:55:56,039 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0, entries=150, sequenceid=234, filesize=11.8 K 2023-04-13 20:55:56,040 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 471ms, sequenceid=234, compaction requested=false 2023-04-13 20:55:56,040 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:56,040 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,040 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12eee8bc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,053 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,053 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,053 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,056 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,057 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,062 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,062 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,063 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,063 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,063 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:56,146 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:55:56,146 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:56,146 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:56,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:56,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:56,147 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:56,147 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:56,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/32ec429d8b1d4c069390d47584874baa 2023-04-13 20:55:56,163 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,173 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e9b2e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,181 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,187 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,187 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,191 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,194 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,194 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,194 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419356220, max timestamp=1681419356244Num uniques 859; 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419356220, max timestamp=1681419356244Num uniques 859; 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419356220, max timestamp=1681419356244Num uniques 859; 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:56,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:56,294 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,298 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18317eb0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,309 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,309 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,309 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,312 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,312 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,312 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,320 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,320 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419416331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419416333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419416341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,420 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,424 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,425 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,427 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,427 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,427 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,427 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,429 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21588cf8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,448 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419416443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419416445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,452 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,452 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,454 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,457 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,462 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,462 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,462 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,462 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,462 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,463 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,463 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,463 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,563 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5b7bff0df9c749819bf44bde47b6f22f 2023-04-13 20:55:56,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,568 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,574 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,579 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,580 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,581 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,581 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c8d8b94db3e544a18b155cc1f8cb45bb 2023-04-13 20:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,582 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44163b67[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,585 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,585 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/32ec429d8b1d4c069390d47584874baa as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa 2023-04-13 20:55:56,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,588 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa, entries=200, sequenceid=247, filesize=14.1 K 2023-04-13 20:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5b7bff0df9c749819bf44bde47b6f22f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f 2023-04-13 20:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,591 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,591 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,591 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f, entries=150, sequenceid=247, filesize=11.8 K 2023-04-13 20:55:56,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c8d8b94db3e544a18b155cc1f8cb45bb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb 2023-04-13 20:55:56,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,597 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,598 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb, entries=150, sequenceid=247, filesize=11.8 K 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b4116c26d7fbdc1e912f851d9767a8d2 in 453ms, sequenceid=247, compaction requested=true 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,599 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:56,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,599 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:56,599 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:56,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:56,599 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,600 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,600 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,600 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,600 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,600 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:56,600 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:55:56,600 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,600 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8912cff09a3f4c6dbc5686f68df7b48b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=37.9 K 2023-04-13 20:55:56,600 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:56,601 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:55:56,601 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,601 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b06f48a4484a4702a041964be1cf48db, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=35.6 K 2023-04-13 20:55:56,602 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 8912cff09a3f4c6dbc5686f68df7b48b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:56,602 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b06f48a4484a4702a041964be1cf48db, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:56,603 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e2ab2b2830344003bf198a6122848423, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681419355179 2023-04-13 20:55:56,603 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 433e50d98a8e43e0a9f04e9adfe162bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681419355179 2023-04-13 20:55:56,603 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 32ec429d8b1d4c069390d47584874baa, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355801 2023-04-13 20:55:56,603 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5b7bff0df9c749819bf44bde47b6f22f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355812 2023-04-13 20:55:56,609 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:56,611 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:56,625 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/34663800782345888dca9ec6c8292454 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/34663800782345888dca9ec6c8292454 2023-04-13 20:55:56,629 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 34663800782345888dca9ec6c8292454(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:56,629 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:56,629 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419356599; duration=0sec 2023-04-13 20:55:56,629 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:56,629 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:55:56,630 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:55:56,630 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:55:56,630 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,630 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/02503561bbe9405e877f0fab1173cc17, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=35.6 K 2023-04-13 20:55:56,631 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 02503561bbe9405e877f0fab1173cc17, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1681419355092 2023-04-13 20:55:56,631 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a1c9eb96b6c24bc09518ca38ae6001f0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1681419355179 2023-04-13 20:55:56,631 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c8d8b94db3e544a18b155cc1f8cb45bb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355812 2023-04-13 20:55:56,638 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#477 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:56,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:56,654 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-13 20:55:56,654 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:56,654 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:56,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:56,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:56,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:56,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:56,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/cb1de79d21ac47278e0bba05fbb17452 2023-04-13 20:55:56,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/13bad8f6df054e62a46729bc54b46277 2023-04-13 20:55:56,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419416676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419416678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,700 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,703 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,706 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,706 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,706 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,706 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,706 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,707 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37526f20[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,713 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,713 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,718 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,718 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,720 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,721 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,724 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,725 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,725 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419416786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419416787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,825 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,828 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,829 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,835 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,836 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,837 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,837 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,838 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24854384[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,840 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,842 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,842 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,844 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,844 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,848 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,850 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,851 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,851 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,851 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,851 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,951 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:56,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:56,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:56,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:56,954 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:56,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:56,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:56,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,955 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,960 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:56,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:56,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:56,960 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:56,960 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:56,960 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:56,960 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:56,960 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:56,960 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,960 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:56,960 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:56,961 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:56,961 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:56,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:56,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41df24a0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:56,965 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:56,965 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,966 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,966 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:56,966 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:56,966 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:56,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:56,967 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,967 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:56,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,967 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:56,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:56,970 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:56,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:56,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:56,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:56,970 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:56,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:56,972 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,972 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:56,975 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:56,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,978 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:56,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:56,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:56,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:56,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:56,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:56,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:56,979 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:56,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:56,979 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:56,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419416994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:56,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:56,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419416994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,026 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/b1cec8ed53da48d79453ce418bd189b1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1cec8ed53da48d79453ce418bd189b1 2023-04-13 20:55:57,031 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into b1cec8ed53da48d79453ce418bd189b1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:57,031 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,031 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419356599; duration=0sec 2023-04-13 20:55:57,031 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:57,049 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/7f64c68edf944ab5bac98bfca47fbf61 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/7f64c68edf944ab5bac98bfca47fbf61 2023-04-13 20:55:57,053 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 7f64c68edf944ab5bac98bfca47fbf61(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:57,053 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,053 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=13, startTime=1681419356599; duration=0sec 2023-04-13 20:55:57,053 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:57,079 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:57,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:57,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:57,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0d6a4994eac342a296a5985162882a53 2023-04-13 20:55:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:57,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:57,082 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:57,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:57,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:57,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,084 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:57,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:57,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:57,086 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,086 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:57,086 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:57,086 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/cb1de79d21ac47278e0bba05fbb17452 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:57,086 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:57,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,089 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:57,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452, entries=200, sequenceid=284, filesize=14.3 K 2023-04-13 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/13bad8f6df054e62a46729bc54b46277 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277 2023-04-13 20:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:57,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34e7f2ff[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:57,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:57,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:57,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:57,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:57,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:57,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:57,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,095 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,095 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:57,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277, entries=150, sequenceid=284, filesize=11.9 K 2023-04-13 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:57,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:57,098 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:57,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0d6a4994eac342a296a5985162882a53 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53 2023-04-13 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:57,098 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:57,098 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:57,100 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,100 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,100 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,101 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,103 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,104 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53, entries=150, sequenceid=284, filesize=11.9 K 2023-04-13 20:55:57,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 451ms, sequenceid=284, compaction requested=false 2023-04-13 20:55:57,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,105 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:57,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,106 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:57,106 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:57,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,206 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:57,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:57,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:57,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:57,209 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:57,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:57,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,211 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:57,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:57,212 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:57,212 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:57,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:57,213 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:57,213 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,220 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/8dc4df74af294212b799f7595ee3d25f 2023-04-13 20:55:57,234 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/635656d5c1d54a04b7e470d4c3c67f09 2023-04-13 20:55:57,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:57,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419357333, max timestamp=1681419357356Num uniques 859; 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419357333, max timestamp=1681419357356Num uniques 859; 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419357333, max timestamp=1681419357356Num uniques 859; 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:57,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:55:57,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419417398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419417400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419417404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419417510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419417510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419417511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,650 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b96c85c64efe435e97f0af8d4db6c5cd 2023-04-13 20:55:57,653 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/8dc4df74af294212b799f7595ee3d25f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f 2023-04-13 20:55:57,657 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f, entries=150, sequenceid=296, filesize=11.9 K 2023-04-13 20:55:57,658 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/635656d5c1d54a04b7e470d4c3c67f09 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09 2023-04-13 20:55:57,661 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09, entries=150, sequenceid=296, filesize=11.9 K 2023-04-13 20:55:57,662 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b96c85c64efe435e97f0af8d4db6c5cd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd 2023-04-13 20:55:57,665 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd, entries=150, sequenceid=296, filesize=11.9 K 2023-04-13 20:55:57,665 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b4116c26d7fbdc1e912f851d9767a8d2 in 452ms, sequenceid=296, compaction requested=true 2023-04-13 20:55:57,665 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,666 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:57,666 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:57,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,669 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:57,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10e8b02c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:57,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:57,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:57,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:57,671 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:57,672 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:57,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,673 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:57,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:57,675 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:57,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:57,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:57,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:57,675 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:57,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,678 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,679 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:57,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,682 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:57,682 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:57,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,682 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:57,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:57,721 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-13 20:55:57,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419357395, max timestamp=1681419357720Num uniques 1018; 2023-04-13 20:55:57,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:57,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419357395, max timestamp=1681419357720Num uniques 1018; 2023-04-13 20:55:57,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419357395, max timestamp=1681419357720Num uniques 1018; 2023-04-13 20:55:57,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:57,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:55:57,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:57,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:57,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:57,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:57,724 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:57,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:57,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/40484352889f4c90adccb6a1e2a7be45 2023-04-13 20:55:57,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c07647b43cd34cd090cf5b79cfbe3a56 2023-04-13 20:55:57,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/309557bd4f894221bcdc9f91dba0d3fb 2023-04-13 20:55:57,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/40484352889f4c90adccb6a1e2a7be45 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45 2023-04-13 20:55:57,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45, entries=200, sequenceid=335, filesize=14.3 K 2023-04-13 20:55:57,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c07647b43cd34cd090cf5b79cfbe3a56 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56 2023-04-13 20:55:57,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56, entries=150, sequenceid=335, filesize=11.9 K 2023-04-13 20:55:57,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/309557bd4f894221bcdc9f91dba0d3fb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb 2023-04-13 20:55:57,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb, entries=150, sequenceid=335, filesize=11.9 K 2023-04-13 20:55:57,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=107.34 KB/109920 for b4116c26d7fbdc1e912f851d9767a8d2 in 46ms, sequenceid=335, compaction requested=true 2023-04-13 20:55:57,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:57,767 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:57,767 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:57,767 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:57,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:57,769 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53918 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:57,769 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:55:57,769 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,769 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/34663800782345888dca9ec6c8292454, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=52.7 K 2023-04-13 20:55:57,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:57,769 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:57,770 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 34663800782345888dca9ec6c8292454, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355812 2023-04-13 20:55:57,769 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,770 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:57,770 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:57,770 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:57,770 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cb1de79d21ac47278e0bba05fbb17452, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681419356285 2023-04-13 20:55:57,770 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1cec8ed53da48d79453ce418bd189b1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=47.9 K 2023-04-13 20:55:57,771 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 8dc4df74af294212b799f7595ee3d25f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681419356663 2023-04-13 20:55:57,771 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b1cec8ed53da48d79453ce418bd189b1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355812 2023-04-13 20:55:57,771 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 40484352889f4c90adccb6a1e2a7be45, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:57,771 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 13bad8f6df054e62a46729bc54b46277, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681419356285 2023-04-13 20:55:57,772 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 635656d5c1d54a04b7e470d4c3c67f09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681419356663 2023-04-13 20:55:57,773 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c07647b43cd34cd090cf5b79cfbe3a56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:57,781 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:57,783 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:57,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:57,789 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:57,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:57,792 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:57,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:57,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:57,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,792 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,794 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:57,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:57,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:57,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:57,802 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,802 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:57,802 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:57,802 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419417788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419417789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419417803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:57,811 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:57,813 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,815 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e1bfff7a36134910b799ded591415bf9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e1bfff7a36134910b799ded591415bf9 2023-04-13 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:57,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eff48d6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:57,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:57,816 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,820 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/35bf466590484c5490d581378a5484b1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/35bf466590484c5490d581378a5484b1 2023-04-13 20:55:57,821 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,822 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into e1bfff7a36134910b799ded591415bf9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:57,822 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:57,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:57,822 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=12, startTime=1681419357767; duration=0sec 2023-04-13 20:55:57,822 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:57,822 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:57,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:57,826 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,827 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49038 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:57,827 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:55:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,827 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,827 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/7f64c68edf944ab5bac98bfca47fbf61, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=47.9 K 2023-04-13 20:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,828 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7f64c68edf944ab5bac98bfca47fbf61, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1681419355812 2023-04-13 20:55:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,828 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0d6a4994eac342a296a5985162882a53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1681419356285 2023-04-13 20:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,829 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b96c85c64efe435e97f0af8d4db6c5cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1681419356663 2023-04-13 20:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,829 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 309557bd4f894221bcdc9f91dba0d3fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:57,829 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:57,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:57,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:57,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:57,830 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:57,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,833 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:57,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:57,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:57,836 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:57,835 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,836 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:57,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,837 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into 35bf466590484c5490d581378a5484b1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:57,837 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:57,838 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=12, startTime=1681419357767; duration=0sec 2023-04-13 20:55:57,838 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:57,841 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:57,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419417904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419417904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:57,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419417909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,936 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:57,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:57,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:57,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:57,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:57,939 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:57,939 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,939 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:57,939 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:57,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,940 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,943 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:57,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:57,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:57,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:57,945 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,945 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:57,945 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:57,945 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:57,945 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,947 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263d858d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:57,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:57,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,950 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:57,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:57,951 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:57,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:57,956 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:57,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:57,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:57,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:57,956 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:57,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:57,959 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:57,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,959 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,961 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:57,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:57,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:57,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:57,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,963 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:57,963 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:57,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:57,963 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:57,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:57,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:57,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:57,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:57,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:57,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,063 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:58,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:58,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:58,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:58,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:58,067 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:58,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:58,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:58,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,069 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,069 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:58,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:58,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:58,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:58,071 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,071 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:58,071 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:58,071 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:58,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:58,072 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,072 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,073 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:58,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d46ebbf[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:58,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:58,074 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,076 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:58,076 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:58,077 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,077 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:58,078 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:58,080 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:58,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:58,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:58,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:58,080 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:58,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:58,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:58,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:58,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,093 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:58,093 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:58,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40134 deadline: 1681419418109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419418110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419418116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,193 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:58,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:58,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/cb3717c233764334b91eb7611c02099d 2023-04-13 20:55:58,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:58,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:58,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:58,196 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:58,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:58,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:58,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,201 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:58,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:58,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:58,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:58,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:58,203 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:58,203 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:58,204 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:58,205 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/e194218d13fa473d95c2ef6319729207 2023-04-13 20:55:58,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:58,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@787a47eb[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:58,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:58,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,208 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:58,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:58,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:58,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,213 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/315dd2e6bb5945d2a34f7733e3c6b941 2023-04-13 20:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:58,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:58,216 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:58,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:58,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:58,217 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:58,217 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:58,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/cb3717c233764334b91eb7611c02099d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d 2023-04-13 20:55:58,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,220 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d, entries=250, sequenceid=355, filesize=16.7 K 2023-04-13 20:55:58,222 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,222 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/e194218d13fa473d95c2ef6319729207 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207 2023-04-13 20:55:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,225 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,225 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,226 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:58,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:58,226 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:58,226 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:58,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207, entries=150, sequenceid=355, filesize=11.9 K 2023-04-13 20:55:58,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/315dd2e6bb5945d2a34f7733e3c6b941 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941 2023-04-13 20:55:58,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941, entries=150, sequenceid=355, filesize=11.9 K 2023-04-13 20:55:58,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b4116c26d7fbdc1e912f851d9767a8d2 in 463ms, sequenceid=355, compaction requested=false 2023-04-13 20:55:58,232 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:58,255 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/6358737bd986403b9a0ba6d936e657bc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/6358737bd986403b9a0ba6d936e657bc 2023-04-13 20:55:58,258 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 6358737bd986403b9a0ba6d936e657bc(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:58,258 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:58,258 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=12, startTime=1681419357768; duration=0sec 2023-04-13 20:55:58,258 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:58,327 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:58,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:58,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:58,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:58,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:58,330 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,330 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,331 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,331 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:58,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:58,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:58,334 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:58,334 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:58,334 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,334 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:58,334 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:58,334 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:58,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:58,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:58,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681419358435, max timestamp=1681419358453Num uniques 859; 2023-04-13 20:55:58,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:58,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681419358435, max timestamp=1681419358453Num uniques 859; 2023-04-13 20:55:58,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:58,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1681419358435, max timestamp=1681419358453Num uniques 859; 2023-04-13 20:55:58,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:55:58,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419418488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419418489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419418490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419418597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419418597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419418598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,746 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/5e0010812a844a19af33a4a2f205a432 2023-04-13 20:55:58,755 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/1bb513619b0548b58a6b7a28d3eccba3 2023-04-13 20:55:58,763 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/d4d5a21c35bb46299852c98da39b7842 2023-04-13 20:55:58,772 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/5e0010812a844a19af33a4a2f205a432 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432 2023-04-13 20:55:58,775 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432, entries=150, sequenceid=370, filesize=11.9 K 2023-04-13 20:55:58,776 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/1bb513619b0548b58a6b7a28d3eccba3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3 2023-04-13 20:55:58,779 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3, entries=150, sequenceid=370, filesize=11.9 K 2023-04-13 20:55:58,779 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/d4d5a21c35bb46299852c98da39b7842 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842 2023-04-13 20:55:58,783 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842, entries=150, sequenceid=370, filesize=11.9 K 2023-04-13 20:55:58,783 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for b4116c26d7fbdc1e912f851d9767a8d2 in 449ms, sequenceid=370, compaction requested=true 2023-04-13 20:55:58,783 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:58,784 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:58,786 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,786 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,787 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca719ca[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:58,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:58,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,791 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:58,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:58,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:58,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:58,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,792 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:58,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:58,796 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:58,796 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:58,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:58,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:58,797 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:58,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,802 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,807 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,807 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:58,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:58,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,808 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:58,808 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:58,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:58,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-13 20:55:58,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:55:58,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/39817fba1b934bf4aa59fe93c6691e0f 2023-04-13 20:55:58,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419418826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419418829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419418834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,908 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:58,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:58,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:58,911 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:58,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:58,911 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:58,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,912 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,913 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,913 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:58,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:58,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:58,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:58,915 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,915 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:58,915 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:58,915 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:58,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fcf63e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:58,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:58,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:58,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:58,922 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:58,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:58,925 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:58,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:58,925 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:58,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:58,926 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:58,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:58,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:58,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:58,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:58,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,932 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:58,932 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:58,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:58,932 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:58,933 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:58,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:58,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419418935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419418935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:58,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:58,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419418937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,033 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,035 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,036 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,039 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,039 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,039 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,039 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:59,039 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:59,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:59,042 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:59,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7de44d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:59,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:59,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,045 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:59,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:59,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:59,049 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:59,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:59,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:59,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:59,049 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:59,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,052 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:59,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:59,060 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:59,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,060 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:59,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419419139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40072 deadline: 1681419419140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419419141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,160 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,163 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,166 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,166 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,170 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,170 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,170 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,170 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:59,170 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:59,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6612b736[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:59,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:59,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:59,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:59,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:59,177 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:59,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:59,181 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:59,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:59,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:59,181 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:59,181 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:59,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,186 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,186 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:59,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:59,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,187 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:59,187 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5e9e863978ff4b1199b7eee33026412b 2023-04-13 20:55:59,287 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,290 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,293 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,294 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,295 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,295 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,295 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,295 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,297 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42df9b77[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:59,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:59,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,299 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,299 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:59,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:59,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:59,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:59,303 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:59,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:59,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:59,303 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:59,303 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:59,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,307 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,310 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:59,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:59,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,311 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,311 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:59,311 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:59,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:59,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,411 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,415 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,415 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,417 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,418 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,419 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,419 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,419 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,419 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:59,419 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:59,420 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,420 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,421 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:59,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a3c814[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:59,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:59,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,425 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:59,425 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:59,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:59,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:59,429 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:59,429 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:59,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:59,430 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:59,430 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:59,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,431 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:59,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:59,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:59,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,441 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:59,441 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:59,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419419443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419419448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419419448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,541 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,544 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,546 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,546 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:55:59,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:55:59,551 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,551 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:55:59,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@505f05a1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:55:59,558 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:55:59,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:55:59,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:55:59,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:55:59,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:55:59,563 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:55:59,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:55:59,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:55:59,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:55:59,563 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:55:59,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:55:59,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,565 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:55:59,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:55:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:55:59,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:55:59,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:55:59,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:55:59,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:55:59,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:55:59,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:55:59,571 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:55:59,571 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:55:59,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/2df540eb506b41e88166ec74d730a015 2023-04-13 20:55:59,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/39817fba1b934bf4aa59fe93c6691e0f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f 2023-04-13 20:55:59,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f, entries=250, sequenceid=405, filesize=16.7 K 2023-04-13 20:55:59,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5e9e863978ff4b1199b7eee33026412b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b 2023-04-13 20:55:59,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b, entries=150, sequenceid=405, filesize=11.9 K 2023-04-13 20:55:59,658 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/2df540eb506b41e88166ec74d730a015 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015 2023-04-13 20:55:59,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015, entries=150, sequenceid=405, filesize=11.9 K 2023-04-13 20:55:59,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 850ms, sequenceid=405, compaction requested=true 2023-04-13 20:55:59,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:59,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:59,662 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:59,662 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:59,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:59,662 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:59,663 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:59,664 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:55:59,664 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,664 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e1bfff7a36134910b799ded591415bf9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=57.7 K 2023-04-13 20:55:59,664 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:59,664 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:55:59,664 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,664 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/35bf466590484c5490d581378a5484b1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.2 K 2023-04-13 20:55:59,664 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e1bfff7a36134910b799ded591415bf9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:59,665 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 35bf466590484c5490d581378a5484b1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:59,665 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cb3717c233764334b91eb7611c02099d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681419357750 2023-04-13 20:55:59,665 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e194218d13fa473d95c2ef6319729207, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681419357759 2023-04-13 20:55:59,666 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5e0010812a844a19af33a4a2f205a432, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681419357779 2023-04-13 20:55:59,666 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1bb513619b0548b58a6b7a28d3eccba3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681419357779 2023-04-13 20:55:59,666 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5e9e863978ff4b1199b7eee33026412b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358488 2023-04-13 20:55:59,666 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 39817fba1b934bf4aa59fe93c6691e0f, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358472 2023-04-13 20:55:59,671 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:55:59,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:55:59,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:55:59,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:55:59,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:55:59,675 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:55:59,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:55:59,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:55:59,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,676 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,677 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:55:59,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:55:59,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,677 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:55:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:55:59,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:55:59,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:55:59,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:55:59,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:55:59,679 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:55:59,679 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,679 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:55:59,682 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#499 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:59,684 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:55:59,684 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:55:59,684 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:59,684 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:55:59,684 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:59,686 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#500 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:59,687 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:55:59,687 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:55:59,718 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/9975499c1f54432ba8d26553c37efdae 2023-04-13 20:55:59,720 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e17e2844fa1d46048bf6601b5a4c18fc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e17e2844fa1d46048bf6601b5a4c18fc 2023-04-13 20:55:59,729 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into e17e2844fa1d46048bf6601b5a4c18fc(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:59,729 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:59,729 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=12, startTime=1681419359662; duration=0sec 2023-04-13 20:55:59,729 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:55:59,729 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:55:59,731 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:55:59,731 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:55:59,731 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:55:59,731 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/6358737bd986403b9a0ba6d936e657bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.2 K 2023-04-13 20:55:59,732 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6358737bd986403b9a0ba6d936e657bc, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1681419357395 2023-04-13 20:55:59,733 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 315dd2e6bb5945d2a34f7733e3c6b941, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1681419357759 2023-04-13 20:55:59,733 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d4d5a21c35bb46299852c98da39b7842, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1681419357779 2023-04-13 20:55:59,734 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 2df540eb506b41e88166ec74d730a015, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358488 2023-04-13 20:55:59,740 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#503 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:55:59,752 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/e2cc9a3f92714bd4af3db64d5109b553 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e2cc9a3f92714bd4af3db64d5109b553 2023-04-13 20:55:59,758 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into e2cc9a3f92714bd4af3db64d5109b553(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:55:59,758 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=12, startTime=1681419359662; duration=0sec 2023-04-13 20:55:59,758 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:55:59,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:55:59,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:55:59,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419359901, max timestamp=1681419359922Num uniques 859; 2023-04-13 20:55:59,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:59,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419359901, max timestamp=1681419359922Num uniques 859; 2023-04-13 20:55:59,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:59,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419359901, max timestamp=1681419359922Num uniques 859; 2023-04-13 20:55:59,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:55:59,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419419965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419419966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419419967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419419968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:55:59,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:55:59,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419419974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419420075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419420075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419420075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419420075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419420080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,099 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/08f3b3f457c54cb1bfb2f3cf03de1611 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/08f3b3f457c54cb1bfb2f3cf03de1611 2023-04-13 20:56:00,103 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into 08f3b3f457c54cb1bfb2f3cf03de1611(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:00,103 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:00,103 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=12, startTime=1681419359662; duration=0sec 2023-04-13 20:56:00,103 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:00,144 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/4d09e92c5c984ad48784d4d948dade36 2023-04-13 20:56:00,153 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/59ea15dd8e2440ada76ab443be395353 2023-04-13 20:56:00,156 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/9975499c1f54432ba8d26553c37efdae as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae 2023-04-13 20:56:00,160 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 20:56:00,160 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/4d09e92c5c984ad48784d4d948dade36 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36 2023-04-13 20:56:00,166 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 20:56:00,168 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/59ea15dd8e2440ada76ab443be395353 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353 2023-04-13 20:56:00,171 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353, entries=150, sequenceid=415, filesize=11.9 K 2023-04-13 20:56:00,172 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b4116c26d7fbdc1e912f851d9767a8d2 in 489ms, sequenceid=415, compaction requested=false 2023-04-13 20:56:00,172 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:00,172 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,172 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,175 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2307a39b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,178 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,179 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,180 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,183 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,183 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,183 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,184 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,186 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:00,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,190 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,190 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:00,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-13 20:56:00,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419359938, max timestamp=1681419360288Num uniques 1068; 2023-04-13 20:56:00,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-13 20:56:00,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:00,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:56:00,290 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:00,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:00,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:00,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1681419359938, max timestamp=1681419360288Num uniques 1068; 2023-04-13 20:56:00,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-13 20:56:00,291 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:00,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,296 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,297 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,297 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:00,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,300 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:00,301 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,301 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,302 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/1c7140d14fe444619316f2b41425c7a8 2023-04-13 20:56:00,303 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,304 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,304 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3d90af[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,308 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,313 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/9f236051411849159ecd295567926f27 2023-04-13 20:56:00,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,314 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,320 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,320 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,320 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,322 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,322 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,324 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,328 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,328 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:00,329 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,329 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419420318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419420318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419420327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,340 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419420330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419420331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,429 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,432 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,437 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,438 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:00,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,438 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,439 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,439 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,439 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:00,439 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,440 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419420432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419420432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419420435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70330874[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,447 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,447 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,449 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,449 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419420442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419420442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,453 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,455 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:00,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,455 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,455 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,555 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,559 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,559 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,561 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:00,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,564 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,564 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,564 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:00,564 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,565 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e979de[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,569 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,570 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,574 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,575 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,578 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,583 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,583 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:00,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419420642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419420655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419420655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419420655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,659 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419420655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,683 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,687 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,687 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,688 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,690 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,690 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,690 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,690 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:00,690 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,692 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466c0223[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,694 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,694 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,695 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,695 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,697 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,697 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,700 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,700 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,700 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,701 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,704 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,706 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,706 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,707 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,707 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,738 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/189311bfcbe743bc8f55ef3d6d47a7f5 2023-04-13 20:56:00,741 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/1c7140d14fe444619316f2b41425c7a8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8 2023-04-13 20:56:00,745 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 20:56:00,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/9f236051411849159ecd295567926f27 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27 2023-04-13 20:56:00,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 20:56:00,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/189311bfcbe743bc8f55ef3d6d47a7f5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5 2023-04-13 20:56:00,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5, entries=150, sequenceid=456, filesize=11.9 K 2023-04-13 20:56:00,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~193.98 KB/198640, currentSize=114.05 KB/116790 for b4116c26d7fbdc1e912f851d9767a8d2 in 466ms, sequenceid=456, compaction requested=true 2023-04-13 20:56:00,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:00,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:00,754 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:00,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:00,754 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:00,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:00,755 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:00,755 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:00,755 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,755 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e17e2844fa1d46048bf6601b5a4c18fc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=36.4 K 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e17e2844fa1d46048bf6601b5a4c18fc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358488 2023-04-13 20:56:00,756 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,756 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/08f3b3f457c54cb1bfb2f3cf03de1611, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=36.4 K 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9975499c1f54432ba8d26553c37efdae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681419358825 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 08f3b3f457c54cb1bfb2f3cf03de1611, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358488 2023-04-13 20:56:00,756 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1c7140d14fe444619316f2b41425c7a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681419359938 2023-04-13 20:56:00,757 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4d09e92c5c984ad48784d4d948dade36, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681419358825 2023-04-13 20:56:00,757 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9f236051411849159ecd295567926f27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681419359938 2023-04-13 20:56:00,764 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#508 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:00,764 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:00,807 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,810 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,812 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,812 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,813 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,813 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:56:00,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:00,813 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,814 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:00,814 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,814 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:00,814 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,821 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/934ba698daf5413d8de5699e85c038b9 2023-04-13 20:56:00,831 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cf1cfe4615e24d03979d98a3c7afb02a 2023-04-13 20:56:00,840 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b3347843982e4d1f81163ce99785ffe4 2023-04-13 20:56:00,844 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/934ba698daf5413d8de5699e85c038b9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9 2023-04-13 20:56:00,849 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9, entries=150, sequenceid=476, filesize=11.9 K 2023-04-13 20:56:00,849 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cf1cfe4615e24d03979d98a3c7afb02a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a 2023-04-13 20:56:00,852 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a, entries=150, sequenceid=476, filesize=11.9 K 2023-04-13 20:56:00,853 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b3347843982e4d1f81163ce99785ffe4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4 2023-04-13 20:56:00,858 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4, entries=150, sequenceid=476, filesize=11.9 K 2023-04-13 20:56:00,859 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for b4116c26d7fbdc1e912f851d9767a8d2 in 46ms, sequenceid=476, compaction requested=true 2023-04-13 20:56:00,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:00,859 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,861 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37a55aa9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,863 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,863 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,868 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:00,868 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:00,868 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,868 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,869 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,869 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:00,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:00,873 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:00,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:00,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:00,873 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:00,873 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:00,874 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,874 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,875 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,876 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:00,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:00,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:00,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:00,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,879 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:00,880 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:00,879 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,880 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,880 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:00,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:00,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:00,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:00,972 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:00,980 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:00,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:00,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:00,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:00,987 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:00,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:00,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:00,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,989 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:00,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:00,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:00,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:00,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:00,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:00,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:00,993 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,993 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:00,993 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:00,993 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:00,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:00,995 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:00,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:00,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/3929eee4770e45719127cc65d51edfed 2023-04-13 20:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:00,995 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:00,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:00,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:00,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f3d8481[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:00,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:00,999 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,000 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:01,000 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:01,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,001 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,003 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,003 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5ec7153516ee49bf8ce5d5a1e5bc201b 2023-04-13 20:56:01,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:01,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:01,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:01,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:01,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:01,006 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:01,006 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:01,008 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,008 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,008 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419360997, max timestamp=1681419361006Num uniques 859; 2023-04-13 20:56:01,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:01,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419360997, max timestamp=1681419361006Num uniques 859; 2023-04-13 20:56:01,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:01,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419360997, max timestamp=1681419361006Num uniques 859; 2023-04-13 20:56:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:01,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,012 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/bd4b3e8a25314999b3e8349524bbd2ed 2023-04-13 20:56:01,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,015 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,015 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,015 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:01,015 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/3929eee4770e45719127cc65d51edfed as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed 2023-04-13 20:56:01,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed, entries=250, sequenceid=486, filesize=16.7 K 2023-04-13 20:56:01,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/5ec7153516ee49bf8ce5d5a1e5bc201b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b 2023-04-13 20:56:01,024 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b, entries=150, sequenceid=486, filesize=11.9 K 2023-04-13 20:56:01,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/bd4b3e8a25314999b3e8349524bbd2ed as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed 2023-04-13 20:56:01,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed, entries=150, sequenceid=486, filesize=11.9 K 2023-04-13 20:56:01,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for b4116c26d7fbdc1e912f851d9767a8d2 in 56ms, sequenceid=486, compaction requested=true 2023-04-13 20:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:56:01,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:01,028 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-13 20:56:01,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=80.51 KB heapSize=225.61 KB 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:01,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:01,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ebe12f7956da4840a298660a43afded0 2023-04-13 20:56:01,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419421053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419421053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,063 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419421059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419421061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,072 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419421062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,115 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:01,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:01,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:01,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:01,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:01,119 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:01,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:01,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:01,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,119 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,120 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,120 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:01,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:01,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:01,122 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,122 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:01,122 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:01,122 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:01,122 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:01,123 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:01,125 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,125 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,126 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:01,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c893131[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:01,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:01,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,129 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:01,129 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,129 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,131 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:01,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:01,133 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:01,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:01,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:01,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:01,133 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:01,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,135 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,135 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,138 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:01,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:01,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,142 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,143 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:01,143 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419421164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419421164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419421164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,178 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2 2023-04-13 20:56:01,178 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/87bea1daf0ff48c1b3e2ae100952d047 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/87bea1daf0ff48c1b3e2ae100952d047 2023-04-13 20:56:01,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419421170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419421173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,187 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 87bea1daf0ff48c1b3e2ae100952d047(size=12.7 K), total size for store is 41.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:01,187 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into cc2c0e2bf1c74f669ddb3bd2090d3bb2(size=12.7 K), total size for store is 36.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,187 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419360754; duration=0sec 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-13 20:56:01,187 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419360754; duration=0sec 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:56:01,187 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:01,189 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42256 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:01,189 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:01,189 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,189 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:01,190 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/87bea1daf0ff48c1b3e2ae100952d047, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=41.3 K 2023-04-13 20:56:01,190 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:56:01,190 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,190 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e2cc9a3f92714bd4af3db64d5109b553, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=60.2 K 2023-04-13 20:56:01,190 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 87bea1daf0ff48c1b3e2ae100952d047, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681419359938 2023-04-13 20:56:01,190 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e2cc9a3f92714bd4af3db64d5109b553, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1681419358488 2023-04-13 20:56:01,191 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 59ea15dd8e2440ada76ab443be395353, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1681419358825 2023-04-13 20:56:01,191 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 934ba698daf5413d8de5699e85c038b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681419360313 2023-04-13 20:56:01,191 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 3929eee4770e45719127cc65d51edfed, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681419360963 2023-04-13 20:56:01,191 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 189311bfcbe743bc8f55ef3d6d47a7f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681419359938 2023-04-13 20:56:01,192 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b3347843982e4d1f81163ce99785ffe4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681419360313 2023-04-13 20:56:01,192 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bd4b3e8a25314999b3e8349524bbd2ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681419360965 2023-04-13 20:56:01,198 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:01,201 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#519 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:01,243 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:01,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:01,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:01,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:01,246 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:01,246 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,246 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:01,246 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:01,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,248 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,248 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,249 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:01,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:01,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:01,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:01,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:01,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:01,253 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,254 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:01,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad25f99[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:01,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:01,255 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:01,256 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,256 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:01,261 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:01,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,261 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:01,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:01,266 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:01,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:01,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:01,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:01,266 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:01,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,268 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,270 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,272 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:01,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:01,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,273 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:01,273 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:01,373 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:01,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:01,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:01,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419421369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419421369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:01,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:01,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419421370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,377 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,379 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,379 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:01,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:01,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:01,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:01,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:01,381 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:01,383 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,383 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,384 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:01,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bf3078[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:01,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:01,385 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,386 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:01,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,386 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,388 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419421381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419421382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:01,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:01,390 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:01,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:01,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:01,390 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:01,391 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:01,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,393 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,396 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,398 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:01,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,398 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:01,398 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:01,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,450 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/ef26a80d9e2c41bdb6931a9ececcc7e1 2023-04-13 20:56:01,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/9d748e12b71a484692494de061b569d6 2023-04-13 20:56:01,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ebe12f7956da4840a298660a43afded0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0 2023-04-13 20:56:01,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0, entries=350, sequenceid=516, filesize=21.4 K 2023-04-13 20:56:01,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/ef26a80d9e2c41bdb6931a9ececcc7e1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1 2023-04-13 20:56:01,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1, entries=150, sequenceid=516, filesize=11.9 K 2023-04-13 20:56:01,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/9d748e12b71a484692494de061b569d6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6 2023-04-13 20:56:01,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6, entries=150, sequenceid=516, filesize=11.9 K 2023-04-13 20:56:01,481 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~245.48 KB/251376, currentSize=93.93 KB/96180 for b4116c26d7fbdc1e912f851d9767a8d2 in 452ms, sequenceid=516, compaction requested=true 2023-04-13 20:56:01,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:56:01,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-13 20:56:01,481 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=5), splitQueue=0 2023-04-13 20:56:01,499 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:01,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:01,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:01,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:01,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:01,501 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:01,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:01,501 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:01,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,502 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:01,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:01,505 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:01,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:01,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:01,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:01,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:01,506 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:01,506 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:01,506 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:01,507 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:01,518 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/1d98ab79c38a4cb996a3625f2a2d5071 2023-04-13 20:56:01,614 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/7de35591e8bd41d2a870c28930e6b32a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/7de35591e8bd41d2a870c28930e6b32a 2023-04-13 20:56:01,615 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c150e8e2c01942bcba21efbf56ce08a0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c150e8e2c01942bcba21efbf56ce08a0 2023-04-13 20:56:01,620 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into c150e8e2c01942bcba21efbf56ce08a0(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:01,620 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 7de35591e8bd41d2a870c28930e6b32a(size=12.8 K), total size for store is 34.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:01,620 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,620 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,620 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=11, startTime=1681419361029; duration=0sec 2023-04-13 20:56:01,620 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419361028; duration=0sec 2023-04-13 20:56:01,621 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=5), splitQueue=0 2023-04-13 20:56:01,621 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=5), splitQueue=0 2023-04-13 20:56:01,621 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:01,622 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:01,622 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.4 K 2023-04-13 20:56:01,622 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cc2c0e2bf1c74f669ddb3bd2090d3bb2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1681419359938 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cf1cfe4615e24d03979d98a3c7afb02a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1681419360313 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 20:56:01,623 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5ec7153516ee49bf8ce5d5a1e5bc201b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681419360965 2023-04-13 20:56:01,624 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ef26a80d9e2c41bdb6931a9ececcc7e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681419361018 2023-04-13 20:56:01,624 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:56:01,624 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:56:01,624 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:01,624 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-13 20:56:01,627 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:56:01,627 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:56:01,627 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:01,633 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#523 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:01,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:01,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:01,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419421698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419421699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419421699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419421700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419421706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419421807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419421807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419421812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419421812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:01,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419421814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,936 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/12debb81c6e64f70a41ac54abd660db9 2023-04-13 20:56:01,946 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/37e92155340a45d3a68dd403e56b5f7f 2023-04-13 20:56:01,950 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/1d98ab79c38a4cb996a3625f2a2d5071 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071 2023-04-13 20:56:01,953 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071, entries=150, sequenceid=535, filesize=11.9 K 2023-04-13 20:56:01,954 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/12debb81c6e64f70a41ac54abd660db9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9 2023-04-13 20:56:01,957 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9, entries=150, sequenceid=535, filesize=11.9 K 2023-04-13 20:56:01,957 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/37e92155340a45d3a68dd403e56b5f7f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f 2023-04-13 20:56:01,962 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f, entries=150, sequenceid=535, filesize=11.9 K 2023-04-13 20:56:01,962 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for b4116c26d7fbdc1e912f851d9767a8d2 in 456ms, sequenceid=535, compaction requested=true 2023-04-13 20:56:01,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:01,962 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:01,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:01,962 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:01,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:01,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,965 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:01,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4495ab82[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:01,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:01,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:01,968 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:01,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,971 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:01,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:01,973 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:01,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:01,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:01,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:01,973 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:01,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:01,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,977 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:01,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:01,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:01,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:01,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:01,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:01,979 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:01,979 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:01,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:01,980 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:01,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:02,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:02,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ab0654f01caa446884b0418a13ea12cb 2023-04-13 20:56:02,042 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/fcf972b171b54505b92586cf813098a2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/fcf972b171b54505b92586cf813098a2 2023-04-13 20:56:02,050 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into fcf972b171b54505b92586cf813098a2(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:02,050 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:02,050 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=12, startTime=1681419361481; duration=0sec 2023-04-13 20:56:02,050 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:02,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419422060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419422061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419422063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419422067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419422068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,080 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:02,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:02,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:02,083 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:02,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:02,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:02,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,086 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,086 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:02,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:02,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:02,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:02,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:02,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:02,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:02,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:02,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:02,087 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,087 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:02,088 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:02,088 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:02,089 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,090 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:02,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41901a98[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:02,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:02,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:02,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:02,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,092 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:02,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:02,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:02,097 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:02,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:02,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:02,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:02,097 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:02,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,100 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,101 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,102 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,107 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,107 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,107 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,107 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:02,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:02,107 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,107 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,107 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:02,107 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:02,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419422169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419422169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419422171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419422174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419422175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,207 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:02,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:02,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:02,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:02,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:02,211 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,211 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,213 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,213 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,213 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,214 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:02,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:02,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:02,214 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:02,214 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,215 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:02,215 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:02,215 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:02,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:02,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:02,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:02,216 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,217 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:02,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3946aba4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:02,218 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:02,218 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,222 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:02,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:02,224 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:02,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:02,224 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:02,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,226 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,226 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,228 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:02,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:02,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:02,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,231 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:02,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,231 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:02,331 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:02,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:02,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:02,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:02,335 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,337 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,337 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,338 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:02,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:02,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:02,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:02,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:02,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:02,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:02,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:02,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,341 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,341 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:02,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f47004[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:02,343 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:02,343 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,344 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:02,344 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:02,344 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,345 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:02,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,346 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:02,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:02,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:02,351 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:02,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:02,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:02,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:02,352 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:02,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:02,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:02,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:02,360 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:02,360 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:02,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419422378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419422379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40088 deadline: 1681419422379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419422379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419422383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,449 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/daec57042fc242a08af13c910cb624a9 2023-04-13 20:56:02,460 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:02,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:02,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:02,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:02,471 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:02,471 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,472 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:02,472 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:02,472 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,473 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,474 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:02,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/9dc905e7d3a7412986622a25059c87ac 2023-04-13 20:56:02,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:02,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:02,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:02,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,476 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:02,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:02,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:02,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:02,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:02,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/ab0654f01caa446884b0418a13ea12cb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb 2023-04-13 20:56:02,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,484 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:02,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:02,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed404c3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:02,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:02,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb, entries=150, sequenceid=553, filesize=11.9 K 2023-04-13 20:56:02,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/daec57042fc242a08af13c910cb624a9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9 2023-04-13 20:56:02,488 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:02,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:02,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,488 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:02,490 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:02,490 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,490 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,490 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9, entries=150, sequenceid=553, filesize=11.9 K 2023-04-13 20:56:02,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:02,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:02,492 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:02,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/9dc905e7d3a7412986622a25059c87ac as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac 2023-04-13 20:56:02,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:02,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:02,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:02,493 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:02,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:02,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:02,495 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:02,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac, entries=150, sequenceid=553, filesize=11.9 K 2023-04-13 20:56:02,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:02,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b4116c26d7fbdc1e912f851d9767a8d2 in 478ms, sequenceid=553, compaction requested=true 2023-04-13 20:56:02,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:02,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:02,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:02,497 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:02,497 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:02,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:02,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:02,499 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:02,499 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:02,499 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,499 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/fcf972b171b54505b92586cf813098a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=36.6 K 2023-04-13 20:56:02,499 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 59428 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:02,500 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fcf972b171b54505b92586cf813098a2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681419361018 2023-04-13 20:56:02,500 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:02,500 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,500 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 12debb81c6e64f70a41ac54abd660db9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681419361052 2023-04-13 20:56:02,500 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/7de35591e8bd41d2a870c28930e6b32a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=58.0 K 2023-04-13 20:56:02,501 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting daec57042fc242a08af13c910cb624a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:02,501 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7de35591e8bd41d2a870c28930e6b32a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681419360965 2023-04-13 20:56:02,501 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ebe12f7956da4840a298660a43afded0, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681419361016 2023-04-13 20:56:02,502 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 1d98ab79c38a4cb996a3625f2a2d5071, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681419361052 2023-04-13 20:56:02,502 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ab0654f01caa446884b0418a13ea12cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:02,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:02,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:02,503 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:02,503 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:02,503 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:02,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:02,508 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#528 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:02,510 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#529 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:02,526 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/3ab1c7d259754689b392a6f83bca6775 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/3ab1c7d259754689b392a6f83bca6775 2023-04-13 20:56:02,530 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into 3ab1c7d259754689b392a6f83bca6775(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:02,530 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:02,530 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419362497; duration=0sec 2023-04-13 20:56:02,530 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:02,530 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:02,531 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:02,531 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:56:02,531 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,532 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c150e8e2c01942bcba21efbf56ce08a0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.5 K 2023-04-13 20:56:02,532 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c150e8e2c01942bcba21efbf56ce08a0, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1681419360965 2023-04-13 20:56:02,532 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9d748e12b71a484692494de061b569d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1681419361018 2023-04-13 20:56:02,533 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 37e92155340a45d3a68dd403e56b5f7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1681419361052 2023-04-13 20:56:02,533 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9dc905e7d3a7412986622a25059c87ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:02,540 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#530 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:02,552 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/21d2b71f24114baa8f556b065aa3b5d4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/21d2b71f24114baa8f556b065aa3b5d4 2023-04-13 20:56:02,556 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 21d2b71f24114baa8f556b065aa3b5d4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:02,556 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:02,556 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=12, startTime=1681419362498; duration=0sec 2023-04-13 20:56:02,556 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:02,603 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:02,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:02,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:02,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:02,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:02,606 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:02,606 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:02,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:02,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,607 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,608 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:02,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:02,608 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,608 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:02,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:02,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:02,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:02,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:02,610 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:02,610 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:02,610 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:02,611 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:02,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:02,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:02,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419422698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419422698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419422702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419422703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419422703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419422805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419422805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419422805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419422809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:02,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419422809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:02,928 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/09917a9e71a842c0a0e8ed43084ff7cc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/09917a9e71a842c0a0e8ed43084ff7cc 2023-04-13 20:56:02,932 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 09917a9e71a842c0a0e8ed43084ff7cc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:02,933 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:02,933 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=12, startTime=1681419362497; duration=0sec 2023-04-13 20:56:02,933 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:03,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419423011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419423012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419423012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,018 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,018 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/01796bdeda2648d382350616f9de767d 2023-04-13 20:56:03,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419423014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40150 deadline: 1681419423015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419423318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419423318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419423318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419423321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419423324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,428 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c5230bafeccf4cabbbd76a030725f7d7 2023-04-13 20:56:03,437 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/63cae507cbde4dfc8cf27d458b3b40de 2023-04-13 20:56:03,441 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/01796bdeda2648d382350616f9de767d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d 2023-04-13 20:56:03,444 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d, entries=150, sequenceid=572, filesize=11.9 K 2023-04-13 20:56:03,444 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c5230bafeccf4cabbbd76a030725f7d7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7 2023-04-13 20:56:03,448 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7, entries=150, sequenceid=572, filesize=11.9 K 2023-04-13 20:56:03,449 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/63cae507cbde4dfc8cf27d458b3b40de as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de 2023-04-13 20:56:03,452 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de, entries=150, sequenceid=572, filesize=11.9 K 2023-04-13 20:56:03,452 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b4116c26d7fbdc1e912f851d9767a8d2 in 842ms, sequenceid=572, compaction requested=false 2023-04-13 20:56:03,452 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:03,452 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:03,453 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,454 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:03,455 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:03,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:03,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f51815b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:03,458 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:03,458 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:03,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:03,460 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:03,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:03,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:03,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:03,462 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:03,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:03,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:03,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:03,464 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:03,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:03,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:03,465 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:03,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:03,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,467 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:03,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:03,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:03,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:03,470 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:03,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:03,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:03,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:03,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:03,471 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:03,471 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:03,571 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:03,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:03,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:03,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:03,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:03,574 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:03,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:03,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:03,574 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:03,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:03,576 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:03,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:03,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,577 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:03,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:03,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:03,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:03,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:03,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:03,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:03,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:03,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:03,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:03,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:03,578 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:03,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:03,578 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:03,578 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:03,579 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:03,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:03,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:03,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419423838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419423843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419423843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419423844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419423846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419423947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,951 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419423947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419423950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419423950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:03,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419423952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:03,990 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/d8b53e0b1e5842768fe2b39c0ccee846 2023-04-13 20:56:04,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419424152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419424153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,159 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419424153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419424155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419424158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,403 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c0c5233dc91942bb99b324d0b9dd374b 2023-04-13 20:56:04,419 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/71d9d3a8fabc452da8455521a2e79048 2023-04-13 20:56:04,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/d8b53e0b1e5842768fe2b39c0ccee846 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846 2023-04-13 20:56:04,429 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846, entries=150, sequenceid=589, filesize=11.9 K 2023-04-13 20:56:04,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c0c5233dc91942bb99b324d0b9dd374b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b 2023-04-13 20:56:04,434 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b, entries=150, sequenceid=589, filesize=11.9 K 2023-04-13 20:56:04,434 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/71d9d3a8fabc452da8455521a2e79048 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048 2023-04-13 20:56:04,439 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048, entries=150, sequenceid=589, filesize=11.9 K 2023-04-13 20:56:04,440 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b4116c26d7fbdc1e912f851d9767a8d2 in 862ms, sequenceid=589, compaction requested=true 2023-04-13 20:56:04,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:04,440 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:04,440 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:04,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,441 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,442 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:04,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c6129e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:04,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:04,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,444 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:04,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,445 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:04,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,446 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:04,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:04,449 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:04,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:04,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:04,449 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:04,449 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:04,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,452 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,452 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,453 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,454 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,457 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,458 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,458 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:04,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:04,458 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,458 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:04,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,458 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:04,458 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:04,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:04,464 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/50ddd11f5ede4504ab3c6ee8a24a13db 2023-04-13 20:56:04,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/45e39945688c4017ba8cc2185d3d6d08 2023-04-13 20:56:04,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419424490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419424490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419424491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419424491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419424498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,558 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:04,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:04,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:04,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:04,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:04,562 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:04,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:04,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:04,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,564 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:04,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:04,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:04,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:04,566 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,566 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:04,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:04,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72624b2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:04,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:04,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:04,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,572 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:04,573 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,573 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:04,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:04,576 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:04,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:04,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:04,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:04,576 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:04,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,578 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,582 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:04,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:04,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,583 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:04,583 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419424604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419424604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419424604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419424604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419424605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,683 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:04,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:04,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:04,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:04,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:04,686 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:04,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:04,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:04,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,686 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,690 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,691 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:04,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:04,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:04,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:04,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:04,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:04,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:04,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:04,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:04,693 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:04,694 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:04,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79aaca89[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:04,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:04,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:04,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,699 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:04,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:04,700 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:04,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:04,702 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:04,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:04,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:04,702 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:04,702 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:04,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,707 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,710 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:04,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:04,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,710 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:04,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,711 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:04,711 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,811 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:04,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419424808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419424808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419424808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419424808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:04,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419424808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:04,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:04,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:04,814 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:04,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:04,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:04,815 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,815 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,818 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,818 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,819 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:04,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:04,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:04,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:04,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:04,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:04,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:04,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:04,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,825 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:04,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a62c7be[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:04,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:04,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,832 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:04,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,836 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,836 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:04,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:04,838 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:04,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:04,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:04,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:04,839 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:04,842 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:04,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,843 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,843 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:04,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,844 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:04,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:04,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:04,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,847 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:04,847 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:04,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:04,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:04,903 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=607 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/ee0f27da442246abbdbc0afaf94d6935 2023-04-13 20:56:04,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/50ddd11f5ede4504ab3c6ee8a24a13db as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db 2023-04-13 20:56:04,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db, entries=250, sequenceid=607, filesize=16.7 K 2023-04-13 20:56:04,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/45e39945688c4017ba8cc2185d3d6d08 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08 2023-04-13 20:56:04,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08, entries=150, sequenceid=607, filesize=11.9 K 2023-04-13 20:56:04,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/ee0f27da442246abbdbc0afaf94d6935 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935 2023-04-13 20:56:04,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935, entries=150, sequenceid=607, filesize=11.9 K 2023-04-13 20:56:04,920 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b4116c26d7fbdc1e912f851d9767a8d2 in 457ms, sequenceid=607, compaction requested=true 2023-04-13 20:56:04,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:04,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:04,920 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:04,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:04,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:56:04,920 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:04,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54680 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:04,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:04,921 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:04,921 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:04,921 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,921 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,922 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/09917a9e71a842c0a0e8ed43084ff7cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=53.4 K 2023-04-13 20:56:04,922 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/3ab1c7d259754689b392a6f83bca6775, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.6 K 2023-04-13 20:56:04,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 09917a9e71a842c0a0e8ed43084ff7cc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:04,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3ab1c7d259754689b392a6f83bca6775, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:04,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 01796bdeda2648d382350616f9de767d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681419362062 2023-04-13 20:56:04,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c5230bafeccf4cabbbd76a030725f7d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681419362062 2023-04-13 20:56:04,922 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d8b53e0b1e5842768fe2b39c0ccee846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681419362696 2023-04-13 20:56:04,923 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c0c5233dc91942bb99b324d0b9dd374b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681419362696 2023-04-13 20:56:04,923 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 50ddd11f5ede4504ab3c6ee8a24a13db, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363836 2023-04-13 20:56:04,923 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 45e39945688c4017ba8cc2185d3d6d08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363845 2023-04-13 20:56:04,932 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#540 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:04,934 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:04,947 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:04,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:04,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:04,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:04,951 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:04,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:04,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:04,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,951 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/5b33d69aecc34fbe891e0aa29bb2221d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5b33d69aecc34fbe891e0aa29bb2221d 2023-04-13 20:56:04,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,952 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:04,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:04,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:04,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:04,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:04,953 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/c385cd8d6a4c49f7a82b64251c1dd63c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c385cd8d6a4c49f7a82b64251c1dd63c 2023-04-13 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:04,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:04,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:04,954 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:04,954 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,954 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:04,954 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:04,955 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:04,957 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into 5b33d69aecc34fbe891e0aa29bb2221d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:04,957 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:04,957 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=12, startTime=1681419364920; duration=0sec 2023-04-13 20:56:04,958 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:04,958 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:04,959 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:04,959 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:56:04,959 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:04,959 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/21d2b71f24114baa8f556b065aa3b5d4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=48.6 K 2023-04-13 20:56:04,959 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 21d2b71f24114baa8f556b065aa3b5d4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1681419361696 2023-04-13 20:56:04,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 63cae507cbde4dfc8cf27d458b3b40de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1681419362062 2023-04-13 20:56:04,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 71d9d3a8fabc452da8455521a2e79048, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=589, earliestPutTs=1681419362696 2023-04-13 20:56:04,960 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ee0f27da442246abbdbc0afaf94d6935, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363845 2023-04-13 20:56:04,966 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into c385cd8d6a4c49f7a82b64251c1dd63c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:04,966 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:04,966 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=12, startTime=1681419364920; duration=0sec 2023-04-13 20:56:04,966 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:04,971 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#543 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:04,978 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/20eddffe48b44d6d82504375e55a298b 2023-04-13 20:56:05,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:05,117 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:05,133 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419425128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419425130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419425131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419425132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419425133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419425235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419425237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419425238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419425238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419425238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,402 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/2f288fa75f3a4086b14fc30595d97252 2023-04-13 20:56:05,409 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/8e70dc03ec52444ebaeeee7d3535bdf3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/8e70dc03ec52444ebaeeee7d3535bdf3 2023-04-13 20:56:05,413 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a63ed96a270c4437abb902e3bfece9e1 2023-04-13 20:56:05,413 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 8e70dc03ec52444ebaeeee7d3535bdf3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:05,413 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:05,414 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=12, startTime=1681419364920; duration=0sec 2023-04-13 20:56:05,414 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:05,417 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/20eddffe48b44d6d82504375e55a298b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b 2023-04-13 20:56:05,421 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b, entries=150, sequenceid=621, filesize=11.9 K 2023-04-13 20:56:05,421 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/2f288fa75f3a4086b14fc30595d97252 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252 2023-04-13 20:56:05,425 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252, entries=150, sequenceid=621, filesize=11.9 K 2023-04-13 20:56:05,426 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a63ed96a270c4437abb902e3bfece9e1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1 2023-04-13 20:56:05,429 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1, entries=150, sequenceid=621, filesize=11.9 K 2023-04-13 20:56:05,430 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b4116c26d7fbdc1e912f851d9767a8d2 in 476ms, sequenceid=621, compaction requested=false 2023-04-13 20:56:05,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:05,430 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:05,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:05,432 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,434 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:05,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:05,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49936a8e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:05,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:05,435 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:05,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,439 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:05,439 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:05,439 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:05,439 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:05,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,444 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:05,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:56:05,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:05,447 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:05,447 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:05,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:05,448 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:05,448 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,450 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,451 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,450 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:05,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:05,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,451 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:05,451 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:05,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419425456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419425457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419425458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419425459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,471 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419425462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,551 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:05,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:05,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:05,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:05,554 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:05,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:05,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:05,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:05,558 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:05,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:05,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:05,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:05,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fa6d4ba[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:05,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:05,566 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:05,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419425563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419425564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:05,568 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:05,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419425566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:05,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,572 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419425568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:05,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419425572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,576 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:05,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:05,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:05,576 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:05,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:05,576 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:05,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,578 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,579 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:05,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:05,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:05,583 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,583 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,584 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,585 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:05,585 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:05,685 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:05,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:05,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:05,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:05,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:05,688 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:05,688 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,689 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,690 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:05,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:05,691 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:05,691 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:05,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:05,692 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:05,692 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:05,695 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,695 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,696 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:05,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5077a745[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:05,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:05,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,698 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:05,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:05,698 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:05,699 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:05,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:05,703 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:05,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:05,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:05,703 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:05,703 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:05,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,704 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,706 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:05,710 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:05,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:05,709 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,710 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:05,710 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:05,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419425769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419425770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419425770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419425778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:05,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419425778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,810 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:05,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:05,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:05,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:05,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:05,813 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:05,813 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:05,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:05,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,815 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,815 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,815 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,815 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:05,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:05,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:05,816 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:05,817 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,817 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:05,817 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:05,817 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:05,817 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:05,818 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,819 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:05,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd4c5c2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:05,821 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:05,821 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,822 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:05,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:05,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:05,824 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:05,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,824 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:05,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:05,827 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:05,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:05,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:05,828 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:05,828 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:05,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:05,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:05,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:05,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:05,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:05,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:05,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:05,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:05,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:05,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:05,834 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,834 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:05,834 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:05,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/fe2d2d3c4e2041bc9d5981a53f73878f 2023-04-13 20:56:05,867 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/2dfad0dd98a34881a48da84b24e8a8dc 2023-04-13 20:56:05,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=646 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/4243c3488f7a44269c19841b8bdb1875 2023-04-13 20:56:05,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/fe2d2d3c4e2041bc9d5981a53f73878f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f 2023-04-13 20:56:05,882 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f, entries=200, sequenceid=646, filesize=14.3 K 2023-04-13 20:56:05,883 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/2dfad0dd98a34881a48da84b24e8a8dc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc 2023-04-13 20:56:05,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc, entries=150, sequenceid=646, filesize=11.9 K 2023-04-13 20:56:05,887 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/4243c3488f7a44269c19841b8bdb1875 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875 2023-04-13 20:56:05,891 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875, entries=150, sequenceid=646, filesize=11.9 K 2023-04-13 20:56:05,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for b4116c26d7fbdc1e912f851d9767a8d2 in 446ms, sequenceid=646, compaction requested=true 2023-04-13 20:56:05,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:05,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:05,892 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:05,892 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:05,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:05,892 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:05,900 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:05,900 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:05,900 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:05,900 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:05,900 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,900 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,900 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5b33d69aecc34fbe891e0aa29bb2221d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=39.2 K 2023-04-13 20:56:05,900 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c385cd8d6a4c49f7a82b64251c1dd63c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=36.9 K 2023-04-13 20:56:05,900 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 5b33d69aecc34fbe891e0aa29bb2221d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363845 2023-04-13 20:56:05,901 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c385cd8d6a4c49f7a82b64251c1dd63c, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363845 2023-04-13 20:56:05,901 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 20eddffe48b44d6d82504375e55a298b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681419364487 2023-04-13 20:56:05,902 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2f288fa75f3a4086b14fc30595d97252, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681419364487 2023-04-13 20:56:05,902 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting fe2d2d3c4e2041bc9d5981a53f73878f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365132 2023-04-13 20:56:05,903 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 2dfad0dd98a34881a48da84b24e8a8dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365445 2023-04-13 20:56:05,912 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#549 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:05,913 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#550 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:05,935 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:05,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:05,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:05,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:05,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:05,948 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:05,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:05,948 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:05,949 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,951 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:05,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:05,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:05,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:05,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:05,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:05,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:05,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:05,956 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/bc51be5ad7a245cb823181f1d56dbf86 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bc51be5ad7a245cb823181f1d56dbf86 2023-04-13 20:56:05,961 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into bc51be5ad7a245cb823181f1d56dbf86(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:05,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:05,961 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419365892; duration=0sec 2023-04-13 20:56:05,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:05,961 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:05,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:05,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:05,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:05,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:05,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:05,964 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:05,964 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:56:05,965 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,965 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/8e70dc03ec52444ebaeeee7d3535bdf3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=36.8 K 2023-04-13 20:56:05,965 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8e70dc03ec52444ebaeeee7d3535bdf3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=607, earliestPutTs=1681419363845 2023-04-13 20:56:05,965 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a63ed96a270c4437abb902e3bfece9e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1681419364487 2023-04-13 20:56:05,966 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4243c3488f7a44269c19841b8bdb1875, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365445 2023-04-13 20:56:05,976 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:05,976 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:05,976 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:05,976 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:05,976 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:56:05,976 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:05,976 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:05,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:05,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:05,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:05,977 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:05,981 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:06,007 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/0a9553794abe4780b4456c261d1c1182 2023-04-13 20:56:06,023 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/24e615a54059435eb71f8ea0a4a10696 2023-04-13 20:56:06,041 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=657 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c823326b8ea04e47b0bf68e3072b4a9d 2023-04-13 20:56:06,045 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/0a9553794abe4780b4456c261d1c1182 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182 2023-04-13 20:56:06,050 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182, entries=150, sequenceid=657, filesize=11.9 K 2023-04-13 20:56:06,050 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/24e615a54059435eb71f8ea0a4a10696 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696 2023-04-13 20:56:06,055 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696, entries=150, sequenceid=657, filesize=11.9 K 2023-04-13 20:56:06,056 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/c823326b8ea04e47b0bf68e3072b4a9d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d 2023-04-13 20:56:06,060 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d, entries=150, sequenceid=657, filesize=11.9 K 2023-04-13 20:56:06,060 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for b4116c26d7fbdc1e912f851d9767a8d2 in 84ms, sequenceid=657, compaction requested=false 2023-04-13 20:56:06,060 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,061 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:06,061 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:06,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:06,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,064 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:06,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:06,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ec10dcc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:06,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:06,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,071 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:06,071 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:06,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,072 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:06,082 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:06,082 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:06,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:06,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:06,085 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:06,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:06,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:06,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:06,085 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:06,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:06,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:06,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:06,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:06,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:06,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:06,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,088 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:56:06,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:06,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:06,090 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:06,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:06,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:06,093 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:06,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:06,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,094 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:06,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:06,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:06,094 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,094 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:06,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:06,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:06,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:06,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/aac5a2e1434949d0885562c428cc854c 2023-04-13 20:56:06,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/a80554805bc5498ea3b7733aa2bccf91 2023-04-13 20:56:06,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419426111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,121 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=669 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a2bd2ea2310544cf9f0a048466c4b952 2023-04-13 20:56:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419426112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419426112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419426114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/aac5a2e1434949d0885562c428cc854c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c 2023-04-13 20:56:06,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419426116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c, entries=200, sequenceid=669, filesize=14.3 K 2023-04-13 20:56:06,129 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/a80554805bc5498ea3b7733aa2bccf91 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91 2023-04-13 20:56:06,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91, entries=150, sequenceid=669, filesize=11.9 K 2023-04-13 20:56:06,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/a2bd2ea2310544cf9f0a048466c4b952 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952 2023-04-13 20:56:06,135 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952, entries=150, sequenceid=669, filesize=11.9 K 2023-04-13 20:56:06,136 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for b4116c26d7fbdc1e912f851d9767a8d2 in 48ms, sequenceid=669, compaction requested=true 2023-04-13 20:56:06,136 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:06,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:56:06,136 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:56:06,194 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:06,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:06,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:06,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:06,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:06,197 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:06,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:06,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:06,197 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:06,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:06,199 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:06,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:06,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:06,199 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:06,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:06,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:06,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:06,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:06,201 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:06,201 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:06,201 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:06,201 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:06,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:06,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:06,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419426236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419426237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419426237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419426238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419426239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419426347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419426350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419426350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419426351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419426351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,387 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/d10b6222a44a464dac7d11b5d163e5a6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d10b6222a44a464dac7d11b5d163e5a6 2023-04-13 20:56:06,392 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into d10b6222a44a464dac7d11b5d163e5a6(size=13.2 K), total size for store is 39.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:06,392 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,392 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419365892; duration=0sec 2023-04-13 20:56:06,392 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-13 20:56:06,392 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-13 20:56:06,393 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-13 20:56:06,393 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-13 20:56:06,393 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:06,393 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:06,394 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40292 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:06,394 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:06,394 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:06,394 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d10b6222a44a464dac7d11b5d163e5a6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=39.3 K 2023-04-13 20:56:06,395 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d10b6222a44a464dac7d11b5d163e5a6, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365445 2023-04-13 20:56:06,395 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0a9553794abe4780b4456c261d1c1182, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681419365456 2023-04-13 20:56:06,395 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aac5a2e1434949d0885562c428cc854c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681419366086 2023-04-13 20:56:06,402 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#559 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:06,402 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/65ce920dc13749869ca262a5b805e0d2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/65ce920dc13749869ca262a5b805e0d2 2023-04-13 20:56:06,407 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 65ce920dc13749869ca262a5b805e0d2(size=13.1 K), total size for store is 36.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:06,407 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,407 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=13, startTime=1681419365892; duration=0sec 2023-04-13 20:56:06,407 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:06,408 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:06,409 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:06,409 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:06,409 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:06,409 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bc51be5ad7a245cb823181f1d56dbf86, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=37.0 K 2023-04-13 20:56:06,409 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bc51be5ad7a245cb823181f1d56dbf86, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365445 2023-04-13 20:56:06,410 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 24e615a54059435eb71f8ea0a4a10696, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681419365456 2023-04-13 20:56:06,410 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a80554805bc5498ea3b7733aa2bccf91, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681419366087 2023-04-13 20:56:06,414 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/a1032dc8f25345cb926a60761d7c1c54 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/a1032dc8f25345cb926a60761d7c1c54 2023-04-13 20:56:06,417 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#560 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:06,422 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into a1032dc8f25345cb926a60761d7c1c54(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:06,422 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,422 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=13, startTime=1681419366136; duration=0sec 2023-04-13 20:56:06,422 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:06,427 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/30af0d9a665743138d9936f019aeda01 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/30af0d9a665743138d9936f019aeda01 2023-04-13 20:56:06,438 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into 30af0d9a665743138d9936f019aeda01(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:06,438 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:06,438 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=13, startTime=1681419366136; duration=0sec 2023-04-13 20:56:06,438 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:06,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419426553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,559 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419426556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419426557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419426557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419426558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,609 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/d981157933bf40799a355f72d197865c 2023-04-13 20:56:06,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419426860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419426862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419426863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419426863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:06,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:06,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419426863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,018 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cd23b206d43342128e49d410ff76e9b8 2023-04-13 20:56:07,028 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=689 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b041b79ae71847e09d00c29f1dc9feb0 2023-04-13 20:56:07,032 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/d981157933bf40799a355f72d197865c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c 2023-04-13 20:56:07,036 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c, entries=150, sequenceid=689, filesize=11.9 K 2023-04-13 20:56:07,037 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cd23b206d43342128e49d410ff76e9b8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8 2023-04-13 20:56:07,040 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8, entries=150, sequenceid=689, filesize=11.9 K 2023-04-13 20:56:07,040 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b041b79ae71847e09d00c29f1dc9feb0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0 2023-04-13 20:56:07,043 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0, entries=150, sequenceid=689, filesize=11.9 K 2023-04-13 20:56:07,044 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for b4116c26d7fbdc1e912f851d9767a8d2 in 843ms, sequenceid=689, compaction requested=true 2023-04-13 20:56:07,044 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:07,044 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,044 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,046 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,048 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b05aa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,049 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,053 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,053 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,055 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,055 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,055 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,055 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,055 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,057 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,057 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,057 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,058 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,065 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,065 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:07,165 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,169 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,172 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,173 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,174 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:56:07,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:07,174 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,175 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:07,175 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,175 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:07,175 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,183 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/517fda6e16554cfa8a27291c64fac68a 2023-04-13 20:56:07,199 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/aee8ddb6adf747a195ca635a2564e65e 2023-04-13 20:56:07,207 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=705 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/847f9d18b57e4805b30062e0ec5ef5d0 2023-04-13 20:56:07,211 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/517fda6e16554cfa8a27291c64fac68a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a 2023-04-13 20:56:07,215 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a, entries=150, sequenceid=705, filesize=11.9 K 2023-04-13 20:56:07,216 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/aee8ddb6adf747a195ca635a2564e65e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e 2023-04-13 20:56:07,219 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e, entries=150, sequenceid=705, filesize=11.9 K 2023-04-13 20:56:07,219 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/847f9d18b57e4805b30062e0ec5ef5d0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0 2023-04-13 20:56:07,223 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0, entries=150, sequenceid=705, filesize=11.9 K 2023-04-13 20:56:07,223 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for b4116c26d7fbdc1e912f851d9767a8d2 in 49ms, sequenceid=705, compaction requested=true 2023-04-13 20:56:07,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:07,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,225 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,226 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,226 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2bc191[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,230 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,230 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,231 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,237 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,237 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,237 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,237 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,237 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,245 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,252 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,252 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:07,253 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,253 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,353 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,356 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,356 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,357 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,359 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:07,360 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,360 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,361 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,362 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,362 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50dbf5c3[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,363 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,364 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,372 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,372 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,378 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,378 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,379 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,379 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,380 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,387 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,388 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,388 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:07,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:07,411 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:07,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419427431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419427432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419367430, max timestamp=1681419367441Num uniques 859; 2023-04-13 20:56:07,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419367430, max timestamp=1681419367441Num uniques 859; 2023-04-13 20:56:07,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:07,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419367430, max timestamp=1681419367441Num uniques 859; 2023-04-13 20:56:07,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:07,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:07,488 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,491 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419427482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419427483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419427488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,492 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,492 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,494 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,494 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,494 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:07,494 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,496 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26022d79[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,497 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,499 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,500 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,500 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,501 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,505 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,506 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,506 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,507 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,507 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,509 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,511 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,512 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,512 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419427544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419427544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419427592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419427592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,600 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419427592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,612 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,615 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,616 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,617 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,617 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,618 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,618 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,619 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:07,619 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,620 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,622 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,622 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cbca46[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,624 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,625 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,625 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,625 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,627 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,627 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,629 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,629 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,630 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,630 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,631 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,632 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,633 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,636 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,637 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,637 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,637 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:07,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,737 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,740 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,742 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,743 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,744 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,745 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,745 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,745 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:07,745 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,745 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,749 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,749 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,750 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d68bda7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,752 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,753 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,753 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,753 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,753 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419427751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419427751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,759 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,759 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,762 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,762 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,762 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,764 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,764 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,764 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,765 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,766 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,770 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,770 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,770 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,770 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,770 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:07,771 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,770 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,771 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,771 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,771 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,771 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,771 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:07,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419427801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419427801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:07,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419427802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=717 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/6f7fb4f5fc934994964c63b286c0253b 2023-04-13 20:56:07,871 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:07,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:07,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:07,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:07,875 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:07,875 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,876 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:07,876 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:07,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,880 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,882 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,882 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,883 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:07,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:07,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:07,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:07,884 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:07,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:07,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:07,885 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:07,885 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:07,886 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,886 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,887 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:07,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42122c24[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:07,888 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:07,888 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,889 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:07,890 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:07,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,890 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,891 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:07,892 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:07,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:07,895 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:07,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:07,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:07,895 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:07,895 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:07,897 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,897 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,897 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:07,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,898 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,899 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:07,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:07,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:07,902 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:07,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:07,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:07,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:07,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:07,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:07,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:07,904 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:07,904 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:08,004 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:08,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:08,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:08,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:08,008 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:08,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:08,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:08,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,008 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,010 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:08,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:08,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,010 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:08,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:08,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:08,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:08,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:08,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:08,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:08,012 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,012 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:08,013 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:08,014 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,014 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:08,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:08,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cca8fa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:08,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:08,016 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,018 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:08,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,018 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:08,018 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:08,019 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:08,020 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,020 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:08,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:08,023 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:08,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:08,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:08,023 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:08,023 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:08,024 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,024 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,025 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:08,025 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:08,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,026 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,028 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,028 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:08,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:08,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:08,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:08,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,029 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:08,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:08,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:08,029 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:08,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419428061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419428061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419428106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419428108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419428108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,130 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:08,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:08,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:08,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:08,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:08,133 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:08,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:08,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:08,133 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,134 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,136 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:08,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:08,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,136 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:08,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:08,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:08,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:08,137 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:08,138 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:08,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,140 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,140 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:08,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c2d5ff[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:08,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:08,142 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,143 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:08,143 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:08,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,143 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:08,144 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:08,145 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:08,149 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:08,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:08,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:08,149 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:08,149 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:08,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,151 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:08,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,152 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:08,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:08,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:08,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:08,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:08,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:08,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:08,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:08,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:08,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:08,156 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:08,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,157 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:08,157 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:08,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:08,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:08,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=717 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/ab771dc214674039805b51a42a9795ca 2023-04-13 20:56:08,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=717 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0c7faa7be56543a28f92b245eff23f95 2023-04-13 20:56:08,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/6f7fb4f5fc934994964c63b286c0253b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b 2023-04-13 20:56:08,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b, entries=250, sequenceid=717, filesize=16.7 K 2023-04-13 20:56:08,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/ab771dc214674039805b51a42a9795ca as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca 2023-04-13 20:56:08,252 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca, entries=150, sequenceid=717, filesize=11.9 K 2023-04-13 20:56:08,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/0c7faa7be56543a28f92b245eff23f95 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95 2023-04-13 20:56:08,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95, entries=150, sequenceid=717, filesize=11.9 K 2023-04-13 20:56:08,257 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:08,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:08,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for b4116c26d7fbdc1e912f851d9767a8d2 in 849ms, sequenceid=717, compaction requested=true 2023-04-13 20:56:08,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:08,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:08,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:08,258 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:08,258 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:56:08,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:56:08,259 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55020 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:08,259 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/A is initiating minor compaction (all files) 2023-04-13 20:56:08,259 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/A in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,259 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/a1032dc8f25345cb926a60761d7c1c54, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=53.7 K 2023-04-13 20:56:08,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:08,260 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a1032dc8f25345cb926a60761d7c1c54, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681419366087 2023-04-13 20:56:08,260 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d981157933bf40799a355f72d197865c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1681419366107 2023-04-13 20:56:08,260 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74376 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:56:08,260 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/C is initiating minor compaction (all files) 2023-04-13 20:56:08,260 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/C in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,261 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/65ce920dc13749869ca262a5b805e0d2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=72.6 K 2023-04-13 20:56:08,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:08,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:08,261 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 517fda6e16554cfa8a27291c64fac68a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681419366233 2023-04-13 20:56:08,261 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:08,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:08,261 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:08,261 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 65ce920dc13749869ca262a5b805e0d2, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=646, earliestPutTs=1681419365445 2023-04-13 20:56:08,261 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6f7fb4f5fc934994964c63b286c0253b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=717, earliestPutTs=1681419367372 2023-04-13 20:56:08,262 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting c823326b8ea04e47b0bf68e3072b4a9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=657, earliestPutTs=1681419365456 2023-04-13 20:56:08,262 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting a2bd2ea2310544cf9f0a048466c4b952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681419366087 2023-04-13 20:56:08,262 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b041b79ae71847e09d00c29f1dc9feb0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1681419366107 2023-04-13 20:56:08,263 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 847f9d18b57e4805b30062e0ec5ef5d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681419366233 2023-04-13 20:56:08,263 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0c7faa7be56543a28f92b245eff23f95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=717, earliestPutTs=1681419367372 2023-04-13 20:56:08,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,268 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:08,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:08,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,273 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#A#compaction#569 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:08,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:08,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:08,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:08,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:08,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:08,281 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:08,282 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#C#compaction#570 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:08,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. started... 2023-04-13 20:56:08,285 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:08,285 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:08,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:08,576 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. as already flushing 2023-04-13 20:56:08,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419428604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419428609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40150 deadline: 1681419428614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40112 deadline: 1681419428624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40072 deadline: 1681419428624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,706 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/b1764c9f72644872a3034f96a73e563c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b1764c9f72644872a3034f96a73e563c 2023-04-13 20:56:08,708 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/13d6cd78e0cc40c789420dc0f3d76186 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/13d6cd78e0cc40c789420dc0f3d76186 2023-04-13 20:56:08,711 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/dc7a828727894f16a700e601117ac30b 2023-04-13 20:56:08,711 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/A of b4116c26d7fbdc1e912f851d9767a8d2 into b1764c9f72644872a3034f96a73e563c(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:08,711 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:08,711 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/A, priority=12, startTime=1681419368258; duration=0sec 2023-04-13 20:56:08,712 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:08,712 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:08,713 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50140 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:08,713 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b4116c26d7fbdc1e912f851d9767a8d2/B is initiating minor compaction (all files) 2023-04-13 20:56:08,714 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b4116c26d7fbdc1e912f851d9767a8d2/B in TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:08,714 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/30af0d9a665743138d9936f019aeda01, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp, totalSize=49.0 K 2023-04-13 20:56:08,714 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 30af0d9a665743138d9936f019aeda01, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=669, earliestPutTs=1681419366087 2023-04-13 20:56:08,715 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting cd23b206d43342128e49d410ff76e9b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=689, earliestPutTs=1681419366107 2023-04-13 20:56:08,715 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/C of b4116c26d7fbdc1e912f851d9767a8d2 into 13d6cd78e0cc40c789420dc0f3d76186(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:08,715 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:08,715 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/C, priority=10, startTime=1681419368258; duration=0sec 2023-04-13 20:56:08,715 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:08,715 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aee8ddb6adf747a195ca635a2564e65e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=705, earliestPutTs=1681419366233 2023-04-13 20:56:08,716 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting ab771dc214674039805b51a42a9795ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=717, earliestPutTs=1681419367372 2023-04-13 20:56:08,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419428725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,734 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b4116c26d7fbdc1e912f851d9767a8d2#B#compaction#573 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:08,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419428726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419428929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:08,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:08,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419428935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,112 DEBUG [Thread-3443] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x680e773e to 127.0.0.1:54584 2023-04-13 20:56:09,112 DEBUG [Thread-3443] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,114 DEBUG [Thread-3439] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:54584 2023-04-13 20:56:09,114 DEBUG [Thread-3439] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,116 DEBUG [Thread-3441] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x659c68ab to 127.0.0.1:54584 2023-04-13 20:56:09,116 DEBUG [Thread-3441] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,117 DEBUG [Thread-3445] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b8f652e to 127.0.0.1:54584 2023-04-13 20:56:09,117 DEBUG [Thread-3445] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,120 DEBUG [Thread-3437] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:54584 2023-04-13 20:56:09,120 DEBUG [Thread-3437] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,135 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cdaf2c155a0b4224935c40264204ce27 2023-04-13 20:56:09,144 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/f2f8916f18a24d7685ec9442afc455ed as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/f2f8916f18a24d7685ec9442afc455ed 2023-04-13 20:56:09,148 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b4116c26d7fbdc1e912f851d9767a8d2/B of b4116c26d7fbdc1e912f851d9767a8d2 into f2f8916f18a24d7685ec9442afc455ed(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:09,148 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:09,148 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2., storeName=b4116c26d7fbdc1e912f851d9767a8d2/B, priority=12, startTime=1681419368258; duration=0sec 2023-04-13 20:56:09,148 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:09,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:09,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40134 deadline: 1681419429231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:09,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40088 deadline: 1681419429238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,542 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=752 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/3529b108a1e547a9a54d886719c79990 2023-04-13 20:56:09,546 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/dc7a828727894f16a700e601117ac30b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/dc7a828727894f16a700e601117ac30b 2023-04-13 20:56:09,549 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/dc7a828727894f16a700e601117ac30b, entries=150, sequenceid=752, filesize=11.9 K 2023-04-13 20:56:09,549 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/cdaf2c155a0b4224935c40264204ce27 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cdaf2c155a0b4224935c40264204ce27 2023-04-13 20:56:09,552 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cdaf2c155a0b4224935c40264204ce27, entries=150, sequenceid=752, filesize=11.9 K 2023-04-13 20:56:09,553 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/3529b108a1e547a9a54d886719c79990 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/3529b108a1e547a9a54d886719c79990 2023-04-13 20:56:09,556 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/3529b108a1e547a9a54d886719c79990, entries=150, sequenceid=752, filesize=11.9 K 2023-04-13 20:56:09,557 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for b4116c26d7fbdc1e912f851d9767a8d2 in 1272ms, sequenceid=752, compaction requested=false 2023-04-13 20:56:09,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:09,557 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:09,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:09,558 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:09,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:09,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:09,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:09,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:09,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:09,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a4fd985[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:09,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:09,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,561 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:09,562 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:09,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,562 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:09,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:09,563 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:09,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:09,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:09,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:09,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:09,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:09,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:09,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:09,565 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:09,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,566 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:09,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,567 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:09,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:09,567 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:09,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:09,570 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:09,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:09,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:09,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:09,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:09,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:09,570 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:09,570 DEBUG [Thread-3436] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:09,570 DEBUG [Thread-3436] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:09,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:09,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:09,632 DEBUG [Thread-3432] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51b8df2a to 127.0.0.1:54584 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:09,632 DEBUG [Thread-3432] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:09,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:09,635 DEBUG [Thread-3434] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:54584 2023-04-13 20:56:09,635 DEBUG [Thread-3434] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,641 DEBUG [Thread-3426] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15880ffb to 127.0.0.1:54584 2023-04-13 20:56:09,641 DEBUG [Thread-3426] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,670 DEBUG [Thread-3436] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:09,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:09,738 DEBUG [Thread-3430] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x420f945a to 127.0.0.1:54584 2023-04-13 20:56:09,738 DEBUG [Thread-3430] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,744 DEBUG [Thread-3428] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x052203cb to 127.0.0.1:54584 2023-04-13 20:56:09,744 DEBUG [Thread-3428] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 198 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 87 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 116 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1712 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5136 rows 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1724 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5172 rows 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1710 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5130 rows 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1703 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5109 rows 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(402): scanned 1726 2023-04-13 20:56:09,744 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(403): verified 5177 rows 2023-04-13 20:56:09,745 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:56:09,745 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37705401 to 127.0.0.1:54584 2023-04-13 20:56:09,745 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:09,746 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:56:09,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:56:09,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:09,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:09,752 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419369752"}]},"ts":"1681419369752"} 2023-04-13 20:56:09,753 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:56:09,850 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:56:09,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, UNASSIGN}] 2023-04-13 20:56:09,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, UNASSIGN 2023-04-13 20:56:09,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:09,853 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=b4116c26d7fbdc1e912f851d9767a8d2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:09,853 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419369852"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419369852"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419369852"}]},"ts":"1681419369852"} 2023-04-13 20:56:09,854 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:56:10,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:10,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b4116c26d7fbdc1e912f851d9767a8d2, disabling compactions & flushes 2023-04-13 20:56:10,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:10,039 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e24e12b6bddb4ca583022d977a3883ae 2023-04-13 20:56:10,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:10,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/b1a408e50e194798898859c7fc473466 2023-04-13 20:56:10,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=767 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/87092b4d57e94de79ab50d4389e5d9c2 2023-04-13 20:56:10,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/e24e12b6bddb4ca583022d977a3883ae as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e24e12b6bddb4ca583022d977a3883ae 2023-04-13 20:56:10,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e24e12b6bddb4ca583022d977a3883ae, entries=150, sequenceid=767, filesize=11.9 K 2023-04-13 20:56:10,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/b1a408e50e194798898859c7fc473466 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1a408e50e194798898859c7fc473466 2023-04-13 20:56:10,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1a408e50e194798898859c7fc473466, entries=150, sequenceid=767, filesize=11.9 K 2023-04-13 20:56:10,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/87092b4d57e94de79ab50d4389e5d9c2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/87092b4d57e94de79ab50d4389e5d9c2 2023-04-13 20:56:10,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/87092b4d57e94de79ab50d4389e5d9c2, entries=150, sequenceid=767, filesize=11.9 K 2023-04-13 20:56:10,471 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=26.84 KB/27480 for b4116c26d7fbdc1e912f851d9767a8d2 in 839ms, sequenceid=767, compaction requested=true 2023-04-13 20:56:10,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:10,471 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:10,471 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:10,471 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:10,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:10,471 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:10,471 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. after waiting 0 ms 2023-04-13 20:56:10,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:10,472 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. because compaction request was cancelled 2023-04-13 20:56:10,472 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b4116c26d7fbdc1e912f851d9767a8d2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=A 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=B 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b4116c26d7fbdc1e912f851d9767a8d2, store=C 2023-04-13 20:56:10,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:10,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:10,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/9561d6a05f594d1cbc96e66bc099036d 2023-04-13 20:56:11,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/13fb2897f7a84edba181a56922652e68 2023-04-13 20:56:11,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b1e28da00f344d2b9a3c59b60bac62a5 2023-04-13 20:56:11,699 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/A/9561d6a05f594d1cbc96e66bc099036d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9561d6a05f594d1cbc96e66bc099036d 2023-04-13 20:56:11,702 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9561d6a05f594d1cbc96e66bc099036d, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 20:56:11,702 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/B/13fb2897f7a84edba181a56922652e68 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13fb2897f7a84edba181a56922652e68 2023-04-13 20:56:11,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13fb2897f7a84edba181a56922652e68, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 20:56:11,705 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/.tmp/C/b1e28da00f344d2b9a3c59b60bac62a5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b1e28da00f344d2b9a3c59b60bac62a5 2023-04-13 20:56:11,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b1e28da00f344d2b9a3c59b60bac62a5, entries=150, sequenceid=774, filesize=11.9 K 2023-04-13 20:56:11,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for b4116c26d7fbdc1e912f851d9767a8d2 in 1237ms, sequenceid=774, compaction requested=true 2023-04-13 20:56:11,732 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/f298fbcf80ee47ebaf14f991bf22b67f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8912cff09a3f4c6dbc5686f68df7b48b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/34663800782345888dca9ec6c8292454, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e1bfff7a36134910b799ded591415bf9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e17e2844fa1d46048bf6601b5a4c18fc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/87bea1daf0ff48c1b3e2ae100952d047, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/7de35591e8bd41d2a870c28930e6b32a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/09917a9e71a842c0a0e8ed43084ff7cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5b33d69aecc34fbe891e0aa29bb2221d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d10b6222a44a464dac7d11b5d163e5a6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/a1032dc8f25345cb926a60761d7c1c54, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b] to archive 2023-04-13 20:56:11,733 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:11,734 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b3aeab3fb73a46c884d5814194fad692 2023-04-13 20:56:11,735 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/bac28b76070b489fb9ea78a419aa2a2a 2023-04-13 20:56:11,736 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/67c072c553864608824365bfc579bcc0 2023-04-13 20:56:11,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ffd172ee39d04ba3b1463d12db0eb3c4 2023-04-13 20:56:11,737 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/582f5d012ae5438dbac55f8830776994 2023-04-13 20:56:11,738 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ece2fd786c2e48289592d2483e1ce480 2023-04-13 20:56:11,739 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/4fc2da1df245409b8c35c30e09e01afa 2023-04-13 20:56:11,740 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/f298fbcf80ee47ebaf14f991bf22b67f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/f298fbcf80ee47ebaf14f991bf22b67f 2023-04-13 20:56:11,741 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/143f1bdf011548708a8a6669244d661c 2023-04-13 20:56:11,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fb9deae2c9db4bec9a17c3549f9f6e02 2023-04-13 20:56:11,742 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8912cff09a3f4c6dbc5686f68df7b48b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8912cff09a3f4c6dbc5686f68df7b48b 2023-04-13 20:56:11,743 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e2ab2b2830344003bf198a6122848423 2023-04-13 20:56:11,744 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/32ec429d8b1d4c069390d47584874baa 2023-04-13 20:56:11,745 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/34663800782345888dca9ec6c8292454 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/34663800782345888dca9ec6c8292454 2023-04-13 20:56:11,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb1de79d21ac47278e0bba05fbb17452 2023-04-13 20:56:11,746 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/8dc4df74af294212b799f7595ee3d25f 2023-04-13 20:56:11,747 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/40484352889f4c90adccb6a1e2a7be45 2023-04-13 20:56:11,748 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e1bfff7a36134910b799ded591415bf9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e1bfff7a36134910b799ded591415bf9 2023-04-13 20:56:11,749 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/cb3717c233764334b91eb7611c02099d 2023-04-13 20:56:11,750 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5e0010812a844a19af33a4a2f205a432 2023-04-13 20:56:11,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/39817fba1b934bf4aa59fe93c6691e0f 2023-04-13 20:56:11,751 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e17e2844fa1d46048bf6601b5a4c18fc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e17e2844fa1d46048bf6601b5a4c18fc 2023-04-13 20:56:11,752 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9975499c1f54432ba8d26553c37efdae 2023-04-13 20:56:11,753 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/87bea1daf0ff48c1b3e2ae100952d047 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/87bea1daf0ff48c1b3e2ae100952d047 2023-04-13 20:56:11,754 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1c7140d14fe444619316f2b41425c7a8 2023-04-13 20:56:11,755 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/934ba698daf5413d8de5699e85c038b9 2023-04-13 20:56:11,756 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/3929eee4770e45719127cc65d51edfed 2023-04-13 20:56:11,757 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/7de35591e8bd41d2a870c28930e6b32a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/7de35591e8bd41d2a870c28930e6b32a 2023-04-13 20:56:11,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ebe12f7956da4840a298660a43afded0 2023-04-13 20:56:11,758 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/1d98ab79c38a4cb996a3625f2a2d5071 2023-04-13 20:56:11,759 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/09917a9e71a842c0a0e8ed43084ff7cc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/09917a9e71a842c0a0e8ed43084ff7cc 2023-04-13 20:56:11,760 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/ab0654f01caa446884b0418a13ea12cb 2023-04-13 20:56:11,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/01796bdeda2648d382350616f9de767d 2023-04-13 20:56:11,761 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d8b53e0b1e5842768fe2b39c0ccee846 2023-04-13 20:56:11,762 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/50ddd11f5ede4504ab3c6ee8a24a13db 2023-04-13 20:56:11,763 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5b33d69aecc34fbe891e0aa29bb2221d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/5b33d69aecc34fbe891e0aa29bb2221d 2023-04-13 20:56:11,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/20eddffe48b44d6d82504375e55a298b 2023-04-13 20:56:11,764 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/fe2d2d3c4e2041bc9d5981a53f73878f 2023-04-13 20:56:11,765 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d10b6222a44a464dac7d11b5d163e5a6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d10b6222a44a464dac7d11b5d163e5a6 2023-04-13 20:56:11,766 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/0a9553794abe4780b4456c261d1c1182 2023-04-13 20:56:11,767 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/aac5a2e1434949d0885562c428cc854c 2023-04-13 20:56:11,768 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/a1032dc8f25345cb926a60761d7c1c54 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/a1032dc8f25345cb926a60761d7c1c54 2023-04-13 20:56:11,769 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/d981157933bf40799a355f72d197865c 2023-04-13 20:56:11,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/517fda6e16554cfa8a27291c64fac68a 2023-04-13 20:56:11,770 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/6f7fb4f5fc934994964c63b286c0253b 2023-04-13 20:56:11,773 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e77c66fa8b1041ac9d409b17008e3a43, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b06f48a4484a4702a041964be1cf48db, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1cec8ed53da48d79453ce418bd189b1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/35bf466590484c5490d581378a5484b1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/08f3b3f457c54cb1bfb2f3cf03de1611, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/fcf972b171b54505b92586cf813098a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/3ab1c7d259754689b392a6f83bca6775, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c385cd8d6a4c49f7a82b64251c1dd63c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bc51be5ad7a245cb823181f1d56dbf86, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/30af0d9a665743138d9936f019aeda01, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca] to archive 2023-04-13 20:56:11,774 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:11,775 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9da682cd17f4447490eb89ec730b9781 2023-04-13 20:56:11,776 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/6ec1046aa4674acf9fc8aca0d11a5a68 2023-04-13 20:56:11,777 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/25ae632af91e4daa905fe04e1e8cb673 2023-04-13 20:56:11,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12637c40d96940ef86ead80b09ead397 2023-04-13 20:56:11,778 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1145ad3bc154436d8962aafc51bac2f4 2023-04-13 20:56:11,779 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/037b7e21b9c1465b966061331f61d29c 2023-04-13 20:56:11,780 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e77c66fa8b1041ac9d409b17008e3a43 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e77c66fa8b1041ac9d409b17008e3a43 2023-04-13 20:56:11,781 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c9b52998fee7466db3949901f9f0a7f6 2023-04-13 20:56:11,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/775e93b595ba43caaeb35f8cb36bf5a6 2023-04-13 20:56:11,782 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b06f48a4484a4702a041964be1cf48db to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b06f48a4484a4702a041964be1cf48db 2023-04-13 20:56:11,783 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bf5a65dde4cb46fcaa5bd886ac91ecc8 2023-04-13 20:56:11,784 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/433e50d98a8e43e0a9f04e9adfe162bb 2023-04-13 20:56:11,785 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1cec8ed53da48d79453ce418bd189b1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1cec8ed53da48d79453ce418bd189b1 2023-04-13 20:56:11,786 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5b7bff0df9c749819bf44bde47b6f22f 2023-04-13 20:56:11,787 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13bad8f6df054e62a46729bc54b46277 2023-04-13 20:56:11,788 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/635656d5c1d54a04b7e470d4c3c67f09 2023-04-13 20:56:11,789 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/35bf466590484c5490d581378a5484b1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/35bf466590484c5490d581378a5484b1 2023-04-13 20:56:11,790 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c07647b43cd34cd090cf5b79cfbe3a56 2023-04-13 20:56:11,791 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/e194218d13fa473d95c2ef6319729207 2023-04-13 20:56:11,792 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/1bb513619b0548b58a6b7a28d3eccba3 2023-04-13 20:56:11,793 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/08f3b3f457c54cb1bfb2f3cf03de1611 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/08f3b3f457c54cb1bfb2f3cf03de1611 2023-04-13 20:56:11,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5e9e863978ff4b1199b7eee33026412b 2023-04-13 20:56:11,794 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/4d09e92c5c984ad48784d4d948dade36 2023-04-13 20:56:11,795 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cc2c0e2bf1c74f669ddb3bd2090d3bb2 2023-04-13 20:56:11,796 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/9f236051411849159ecd295567926f27 2023-04-13 20:56:11,797 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cf1cfe4615e24d03979d98a3c7afb02a 2023-04-13 20:56:11,798 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/5ec7153516ee49bf8ce5d5a1e5bc201b 2023-04-13 20:56:11,799 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/fcf972b171b54505b92586cf813098a2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/fcf972b171b54505b92586cf813098a2 2023-04-13 20:56:11,800 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ef26a80d9e2c41bdb6931a9ececcc7e1 2023-04-13 20:56:11,801 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/12debb81c6e64f70a41ac54abd660db9 2023-04-13 20:56:11,802 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/3ab1c7d259754689b392a6f83bca6775 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/3ab1c7d259754689b392a6f83bca6775 2023-04-13 20:56:11,803 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/daec57042fc242a08af13c910cb624a9 2023-04-13 20:56:11,804 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c5230bafeccf4cabbbd76a030725f7d7 2023-04-13 20:56:11,805 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c0c5233dc91942bb99b324d0b9dd374b 2023-04-13 20:56:11,806 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c385cd8d6a4c49f7a82b64251c1dd63c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/c385cd8d6a4c49f7a82b64251c1dd63c 2023-04-13 20:56:11,807 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/45e39945688c4017ba8cc2185d3d6d08 2023-04-13 20:56:11,808 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2f288fa75f3a4086b14fc30595d97252 2023-04-13 20:56:11,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bc51be5ad7a245cb823181f1d56dbf86 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/bc51be5ad7a245cb823181f1d56dbf86 2023-04-13 20:56:11,809 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/2dfad0dd98a34881a48da84b24e8a8dc 2023-04-13 20:56:11,810 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/24e615a54059435eb71f8ea0a4a10696 2023-04-13 20:56:11,811 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/30af0d9a665743138d9936f019aeda01 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/30af0d9a665743138d9936f019aeda01 2023-04-13 20:56:11,812 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/a80554805bc5498ea3b7733aa2bccf91 2023-04-13 20:56:11,813 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cd23b206d43342128e49d410ff76e9b8 2023-04-13 20:56:11,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/aee8ddb6adf747a195ca635a2564e65e 2023-04-13 20:56:11,814 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/ab771dc214674039805b51a42a9795ca 2023-04-13 20:56:11,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e1524e8f7f6d434db2c6879b78057733, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/02503561bbe9405e877f0fab1173cc17, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/7f64c68edf944ab5bac98bfca47fbf61, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/6358737bd986403b9a0ba6d936e657bc, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e2cc9a3f92714bd4af3db64d5109b553, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c150e8e2c01942bcba21efbf56ce08a0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/21d2b71f24114baa8f556b065aa3b5d4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/8e70dc03ec52444ebaeeee7d3535bdf3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/65ce920dc13749869ca262a5b805e0d2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95] to archive 2023-04-13 20:56:11,818 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:11,819 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/782ae9c9a253406f8e568586222d23cd 2023-04-13 20:56:11,820 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d352663878e48b2977a1abedef418c5 2023-04-13 20:56:11,821 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2881a18233514adc98364c6b3d60b2cc 2023-04-13 20:56:11,822 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/38486c5ebbbf401f845c98d80a310833 2023-04-13 20:56:11,823 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c6834b64b533462e8bbd42af57b4e44b 2023-04-13 20:56:11,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/14d0d3ea2f06468990e51d04dafce613 2023-04-13 20:56:11,824 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e1524e8f7f6d434db2c6879b78057733 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e1524e8f7f6d434db2c6879b78057733 2023-04-13 20:56:11,825 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/f625e1ca168b4fc097c786b2f558489b 2023-04-13 20:56:11,826 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e96081a21d7c43aa8f1e247c1af6a4cf 2023-04-13 20:56:11,827 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/02503561bbe9405e877f0fab1173cc17 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/02503561bbe9405e877f0fab1173cc17 2023-04-13 20:56:11,828 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0e81e997450d491fbdd2712d0a056b40 2023-04-13 20:56:11,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a1c9eb96b6c24bc09518ca38ae6001f0 2023-04-13 20:56:11,829 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/7f64c68edf944ab5bac98bfca47fbf61 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/7f64c68edf944ab5bac98bfca47fbf61 2023-04-13 20:56:11,830 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c8d8b94db3e544a18b155cc1f8cb45bb 2023-04-13 20:56:11,831 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0d6a4994eac342a296a5985162882a53 2023-04-13 20:56:11,832 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b96c85c64efe435e97f0af8d4db6c5cd 2023-04-13 20:56:11,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/6358737bd986403b9a0ba6d936e657bc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/6358737bd986403b9a0ba6d936e657bc 2023-04-13 20:56:11,833 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/309557bd4f894221bcdc9f91dba0d3fb 2023-04-13 20:56:11,834 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/315dd2e6bb5945d2a34f7733e3c6b941 2023-04-13 20:56:11,835 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/d4d5a21c35bb46299852c98da39b7842 2023-04-13 20:56:11,836 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e2cc9a3f92714bd4af3db64d5109b553 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/e2cc9a3f92714bd4af3db64d5109b553 2023-04-13 20:56:11,837 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/2df540eb506b41e88166ec74d730a015 2023-04-13 20:56:11,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/59ea15dd8e2440ada76ab443be395353 2023-04-13 20:56:11,838 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/189311bfcbe743bc8f55ef3d6d47a7f5 2023-04-13 20:56:11,839 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b3347843982e4d1f81163ce99785ffe4 2023-04-13 20:56:11,840 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c150e8e2c01942bcba21efbf56ce08a0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c150e8e2c01942bcba21efbf56ce08a0 2023-04-13 20:56:11,841 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/bd4b3e8a25314999b3e8349524bbd2ed 2023-04-13 20:56:11,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9d748e12b71a484692494de061b569d6 2023-04-13 20:56:11,842 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/37e92155340a45d3a68dd403e56b5f7f 2023-04-13 20:56:11,843 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/21d2b71f24114baa8f556b065aa3b5d4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/21d2b71f24114baa8f556b065aa3b5d4 2023-04-13 20:56:11,844 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/9dc905e7d3a7412986622a25059c87ac 2023-04-13 20:56:11,845 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/63cae507cbde4dfc8cf27d458b3b40de 2023-04-13 20:56:11,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/71d9d3a8fabc452da8455521a2e79048 2023-04-13 20:56:11,846 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/8e70dc03ec52444ebaeeee7d3535bdf3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/8e70dc03ec52444ebaeeee7d3535bdf3 2023-04-13 20:56:11,847 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/ee0f27da442246abbdbc0afaf94d6935 2023-04-13 20:56:11,848 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a63ed96a270c4437abb902e3bfece9e1 2023-04-13 20:56:11,849 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/65ce920dc13749869ca262a5b805e0d2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/65ce920dc13749869ca262a5b805e0d2 2023-04-13 20:56:11,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/4243c3488f7a44269c19841b8bdb1875 2023-04-13 20:56:11,850 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/c823326b8ea04e47b0bf68e3072b4a9d 2023-04-13 20:56:11,851 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/a2bd2ea2310544cf9f0a048466c4b952 2023-04-13 20:56:11,852 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b041b79ae71847e09d00c29f1dc9feb0 2023-04-13 20:56:11,853 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/847f9d18b57e4805b30062e0ec5ef5d0 2023-04-13 20:56:11,854 DEBUG [StoreCloser-TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/0c7faa7be56543a28f92b245eff23f95 2023-04-13 20:56:11,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:11,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/recovered.edits/777.seqid, newMaxSeqId=777, maxSeqId=1 2023-04-13 20:56:11,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2. 2023-04-13 20:56:11,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b4116c26d7fbdc1e912f851d9767a8d2: 2023-04-13 20:56:11,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:11,860 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=b4116c26d7fbdc1e912f851d9767a8d2, regionState=CLOSED 2023-04-13 20:56:11,860 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419371860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419371860"}]},"ts":"1681419371860"} 2023-04-13 20:56:11,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-13 20:56:11,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure b4116c26d7fbdc1e912f851d9767a8d2, server=jenkins-hbase4.apache.org,36431,1681419217857 in 2.0070 sec 2023-04-13 20:56:11,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-13 20:56:11,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b4116c26d7fbdc1e912f851d9767a8d2, UNASSIGN in 2.0110 sec 2023-04-13 20:56:11,864 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419371864"}]},"ts":"1681419371864"} 2023-04-13 20:56:11,865 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:56:11,866 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:56:11,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.1190 sec 2023-04-13 20:56:13,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-13 20:56:13,857 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-13 20:56:13,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:56:13,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,859 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 20:56:13,860 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,864 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:13,866 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/recovered.edits] 2023-04-13 20:56:13,868 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9561d6a05f594d1cbc96e66bc099036d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/9561d6a05f594d1cbc96e66bc099036d 2023-04-13 20:56:13,869 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b1764c9f72644872a3034f96a73e563c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/b1764c9f72644872a3034f96a73e563c 2023-04-13 20:56:13,870 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/dc7a828727894f16a700e601117ac30b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/dc7a828727894f16a700e601117ac30b 2023-04-13 20:56:13,871 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e24e12b6bddb4ca583022d977a3883ae to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/A/e24e12b6bddb4ca583022d977a3883ae 2023-04-13 20:56:13,873 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13fb2897f7a84edba181a56922652e68 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/13fb2897f7a84edba181a56922652e68 2023-04-13 20:56:13,874 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1a408e50e194798898859c7fc473466 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/b1a408e50e194798898859c7fc473466 2023-04-13 20:56:13,875 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cdaf2c155a0b4224935c40264204ce27 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/cdaf2c155a0b4224935c40264204ce27 2023-04-13 20:56:13,876 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/f2f8916f18a24d7685ec9442afc455ed to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/B/f2f8916f18a24d7685ec9442afc455ed 2023-04-13 20:56:13,878 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/13d6cd78e0cc40c789420dc0f3d76186 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/13d6cd78e0cc40c789420dc0f3d76186 2023-04-13 20:56:13,878 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/3529b108a1e547a9a54d886719c79990 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/3529b108a1e547a9a54d886719c79990 2023-04-13 20:56:13,879 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/87092b4d57e94de79ab50d4389e5d9c2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/87092b4d57e94de79ab50d4389e5d9c2 2023-04-13 20:56:13,880 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b1e28da00f344d2b9a3c59b60bac62a5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/C/b1e28da00f344d2b9a3c59b60bac62a5 2023-04-13 20:56:13,882 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/recovered.edits/777.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2/recovered.edits/777.seqid 2023-04-13 20:56:13,883 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b4116c26d7fbdc1e912f851d9767a8d2 2023-04-13 20:56:13,883 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:56:13,884 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,885 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:56:13,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:56:13,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:56:13,888 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419373888"}]},"ts":"9223372036854775807"} 2023-04-13 20:56:13,889 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:56:13,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b4116c26d7fbdc1e912f851d9767a8d2, NAME => 'TestAcidGuarantees,,1681419347934.b4116c26d7fbdc1e912f851d9767a8d2.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:56:13,889 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:56:13,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419373889"}]},"ts":"9223372036854775807"} 2023-04-13 20:56:13,890 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:56:13,892 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-04-13 20:56:13,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-13 20:56:13,961 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-13 20:56:13,972 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228 (was 227) - Thread LEAK? -, OpenFileDescriptor=589 (was 526) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=422 (was 369) - SystemLoadAverage LEAK? -, ProcessCount=173 (was 173), AvailableMemoryMB=887 (was 735) - AvailableMemoryMB LEAK? - 2023-04-13 20:56:13,984 INFO [Listener at localhost/42015] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228, OpenFileDescriptor=589, MaxFileDescriptor=60000, SystemLoadAverage=422, ProcessCount=173, AvailableMemoryMB=887 2023-04-13 20:56:13,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:56:13,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:56:13,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:13,988 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-13 20:56:13,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-13 20:56:13,989 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-13 20:56:13,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 20:56:13,990 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:13,991 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d empty. 2023-04-13 20:56:13,991 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:13,991 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:56:13,998 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:56:13,999 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b19b9c1a7b3fa24b26a81e15015b427d, NAME => 'TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp 2023-04-13 20:56:14,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 20:56:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b19b9c1a7b3fa24b26a81e15015b427d, disabling compactions & flushes 2023-04-13 20:56:14,408 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. after waiting 0 ms 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,408 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,408 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:14,410 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-13 20:56:14,411 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419374411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419374411"}]},"ts":"1681419374411"} 2023-04-13 20:56:14,412 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-13 20:56:14,412 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-13 20:56:14,412 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419374412"}]},"ts":"1681419374412"} 2023-04-13 20:56:14,413 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-13 20:56:14,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, ASSIGN}] 2023-04-13 20:56:14,418 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, ASSIGN 2023-04-13 20:56:14,418 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=false 2023-04-13 20:56:14,570 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:14,570 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419374570"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419374570"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419374570"}]},"ts":"1681419374570"} 2023-04-13 20:56:14,571 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:56:14,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 20:56:14,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b19b9c1a7b3fa24b26a81e15015b427d, NAME => 'TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:56:14,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:56:14,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,728 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,730 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A 2023-04-13 20:56:14,730 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A 2023-04-13 20:56:14,730 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:14,730 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName A 2023-04-13 20:56:14,731 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:14,731 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,732 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B 2023-04-13 20:56:14,732 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B 2023-04-13 20:56:14,732 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:14,732 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName B 2023-04-13 20:56:14,733 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:14,733 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,734 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C 2023-04-13 20:56:14,734 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C 2023-04-13 20:56:14,734 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:14,734 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName C 2023-04-13 20:56:14,735 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:14,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:56:14,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:14,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-13 20:56:14,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b19b9c1a7b3fa24b26a81e15015b427d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70052545, jitterRate=0.04386426508426666}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:56:14,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:14,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., pid=56, masterSystemTime=1681419374723 2023-04-13 20:56:14,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:14,744 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:14,744 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419374744"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419374744"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419374744"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419374744"}]},"ts":"1681419374744"} 2023-04-13 20:56:14,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-13 20:56:14,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 in 174 msec 2023-04-13 20:56:14,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-13 20:56:14,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, ASSIGN in 329 msec 2023-04-13 20:56:14,747 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-13 20:56:14,747 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419374747"}]},"ts":"1681419374747"} 2023-04-13 20:56:14,748 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-13 20:56:14,750 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-13 20:56:14,750 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 764 msec 2023-04-13 20:56:15,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-13 20:56:15,093 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-13 20:56:15,094 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:15,099 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:15,100 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:15,101 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:15,102 DEBUG [Listener at localhost/42015] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-13 20:56:15,104 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57942, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-13 20:56:15,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-13 20:56:15,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-13 20:56:15,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:15,516 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:56:15,517 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-13 20:56:15,517 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-13 20:56:15,519 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-13 20:56:15,523 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, REOPEN/MOVE}] 2023-04-13 20:56:15,524 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, REOPEN/MOVE 2023-04-13 20:56:15,524 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:15,524 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419375524"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419375524"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419375524"}]},"ts":"1681419375524"} 2023-04-13 20:56:15,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:56:15,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b19b9c1a7b3fa24b26a81e15015b427d, disabling compactions & flushes 2023-04-13 20:56:15,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:15,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:15,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. after waiting 0 ms 2023-04-13 20:56:15,678 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:15,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-13 20:56:15,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:15,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:15,681 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b19b9c1a7b3fa24b26a81e15015b427d to self. 2023-04-13 20:56:15,683 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,683 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=CLOSED 2023-04-13 20:56:15,683 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419375683"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419375683"}]},"ts":"1681419375683"} 2023-04-13 20:56:15,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-13 20:56:15,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 in 159 msec 2023-04-13 20:56:15,685 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,36431,1681419217857; forceNewPlan=false, retain=true 2023-04-13 20:56:15,836 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:15,836 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419375836"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419375836"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419375836"}]},"ts":"1681419375836"} 2023-04-13 20:56:15,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:56:15,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:15,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b19b9c1a7b3fa24b26a81e15015b427d, NAME => 'TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.', STARTKEY => '', ENDKEY => ''} 2023-04-13 20:56:15,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-13 20:56:15,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,993 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,995 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A 2023-04-13 20:56:15,995 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A 2023-04-13 20:56:15,995 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:15,995 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName A 2023-04-13 20:56:15,997 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:15,997 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:15,998 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B 2023-04-13 20:56:15,998 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B 2023-04-13 20:56:15,998 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:15,998 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName B 2023-04-13 20:56:15,999 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:15,999 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:16,000 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C 2023-04-13 20:56:16,000 DEBUG [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C 2023-04-13 20:56:16,000 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-13 20:56:16,000 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b19b9c1a7b3fa24b26a81e15015b427d columnFamilyName C 2023-04-13 20:56:16,000 INFO [StoreOpener-b19b9c1a7b3fa24b26a81e15015b427d-1] regionserver.HStore(310): Store=b19b9c1a7b3fa24b26a81e15015b427d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-13 20:56:16,001 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:16,001 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:16,002 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:16,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-13 20:56:16,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:16,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b19b9c1a7b3fa24b26a81e15015b427d; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69305770, jitterRate=0.03273645043373108}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-13 20:56:16,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:16,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., pid=61, masterSystemTime=1681419375989 2023-04-13 20:56:16,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:16,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:16,008 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:16,008 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419376008"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681419376008"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681419376008"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681419376008"}]},"ts":"1681419376008"} 2023-04-13 20:56:16,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-13 20:56:16,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 in 172 msec 2023-04-13 20:56:16,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-13 20:56:16,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, REOPEN/MOVE in 487 msec 2023-04-13 20:56:16,013 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-13 20:56:16,013 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 493 msec 2023-04-13 20:56:16,014 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 909 msec 2023-04-13 20:56:16,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-13 20:56:16,016 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,024 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,024 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,028 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,028 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,037 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,038 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,041 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,041 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,047 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,048 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,051 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,052 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,056 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,056 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,063 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,064 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,075 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,076 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:54584 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-13 20:56:16,079 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-13 20:56:16,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:16,086 DEBUG [hconnection-0x50c10315-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,087 DEBUG [hconnection-0x301ee8ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:16,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:16,087 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:16,087 DEBUG [hconnection-0x41a0c4b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:16,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:16,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:16,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:16,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:16,088 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39670, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:16,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:16,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:16,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:16,089 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:16,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:16,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:16,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:16,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:16,092 DEBUG [hconnection-0x44d4c25b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,097 DEBUG [hconnection-0x9fee631-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,097 DEBUG [hconnection-0x277b520b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:16,101 DEBUG [hconnection-0x1cc198a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:16,102 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39658, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,100 DEBUG [hconnection-0x12a33a52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,099 DEBUG [hconnection-0x76244fcd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,100 DEBUG [hconnection-0x9c2bb1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-13 20:56:16,104 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,102 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,104 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39674, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,104 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39686, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,103 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:16,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:16,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:16,106 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39710, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,108 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:16,110 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:16,111 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:16,111 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,112 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39694, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,115 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39722, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-13 20:56:16,115 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-13 20:56:16,117 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:16,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:16,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:16,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:16,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:16,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:16,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:16,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:16,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681419376127, max timestamp=1681419376129Num uniques 859; 2023-04-13 20:56:16,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:16,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1681419376123, max timestamp=1681419376129Num uniques 859; 2023-04-13 20:56:16,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-13 20:56:16,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1681419376127, max timestamp=1681419376129Num uniques 859; 2023-04-13 20:56:16,133 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419376134, max timestamp=1681419376157Num uniques 1068; 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419376134, max timestamp=1681419376157Num uniques 1168; 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419376134, max timestamp=1681419376157Num uniques 1068; 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-04-13 20:56:16,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:16,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419376161, max timestamp=1681419376170Num uniques 1027; 2023-04-13 20:56:16,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419376161, max timestamp=1681419376170Num uniques 1027; 2023-04-13 20:56:16,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:56:16,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:56:16,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419376161, max timestamp=1681419376170Num uniques 1027; 2023-04-13 20:56:16,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:56:16,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419376179, max timestamp=1681419376183Num uniques 1086; 2023-04-13 20:56:16,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419376179, max timestamp=1681419376183Num uniques 1086; 2023-04-13 20:56:16,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:56:16,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419376179, max timestamp=1681419376183Num uniques 1086; 2023-04-13 20:56:16,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:56:16,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:56:16,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419376195, max timestamp=1681419376199Num uniques 1045; 2023-04-13 20:56:16,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 20:56:16,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419376195, max timestamp=1681419376199Num uniques 1045; 2023-04-13 20:56:16,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-13 20:56:16,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376195, max timestamp=1681419376199Num uniques 1045; 2023-04-13 20:56:16,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-04-13 20:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419376204, max timestamp=1681419376214Num uniques 1009; 2023-04-13 20:56:16,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46210, new segment data size=7050, suffix heap size=132904,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=154 2023-04-13 20:56:16,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419376204, max timestamp=1681419376214Num uniques 1054; 2023-04-13 20:56:16,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 20:56:16,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1681419376204, max timestamp=1681419376214Num uniques 1054; 2023-04-13 20:56:16,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-04-13 20:56:16,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1681419376221, max timestamp=1681419376225Num uniques 1013; 2023-04-13 20:56:16,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=9745, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:56:16,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9745, totalHeapSize=22392, min timestamp=1681419376221, max timestamp=1681419376225Num uniques 1113; 2023-04-13 20:56:16,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419376221, max timestamp=1681419376225Num uniques 1013; 2023-04-13 20:56:16,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50965, new segment data size=9745, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-13 20:56:16,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419376241, max timestamp=1681419376245Num uniques 1072; 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419376241, max timestamp=1681419376245Num uniques 1072; 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419376241, max timestamp=1681419376245Num uniques 1072; 2023-04-13 20:56:16,247 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-13 20:56:16,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419376255, max timestamp=1681419376259Num uniques 1031; 2023-04-13 20:56:16,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-13 20:56:16,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419376255, max timestamp=1681419376259Num uniques 1031; 2023-04-13 20:56:16,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-13 20:56:16,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419376255, max timestamp=1681419376259Num uniques 1031; 2023-04-13 20:56:16,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-13 20:56:16,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419376272, max timestamp=1681419376275Num uniques 1090; 2023-04-13 20:56:16,279 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-13 20:56:16,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1681419376272, max timestamp=1681419376275Num uniques 1090; 2023-04-13 20:56:16,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-04-13 20:56:16,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376272, max timestamp=1681419376275Num uniques 1090; 2023-04-13 20:56:16,280 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=6870, suffix heap size=141328,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=150 2023-04-13 20:56:16,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419376287, max timestamp=1681419376290Num uniques 1009; 2023-04-13 20:56:16,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419376287, max timestamp=1681419376290Num uniques 1049; 2023-04-13 20:56:16,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11404, suffix heap size=132904,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=249 2023-04-13 20:56:16,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:56:16,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419376287, max timestamp=1681419376290Num uniques 1049; 2023-04-13 20:56:16,293 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419376300, max timestamp=1681419376304Num uniques 1108; 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419376300, max timestamp=1681419376304Num uniques 1108; 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1681419376300, max timestamp=1681419376304Num uniques 1108; 2023-04-13 20:56:16,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-04-13 20:56:16,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419376316, max timestamp=1681419376319Num uniques 1117; 2023-04-13 20:56:16,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:56:16,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419376316, max timestamp=1681419376319Num uniques 1117; 2023-04-13 20:56:16,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:56:16,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1681419376316, max timestamp=1681419376319Num uniques 1117; 2023-04-13 20:56:16,322 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-04-13 20:56:16,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681419376326, max timestamp=1681419376331Num uniques 1076; 2023-04-13 20:56:16,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:56:16,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681419376326, max timestamp=1681419376331Num uniques 1076; 2023-04-13 20:56:16,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:56:16,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1681419376326, max timestamp=1681419376331Num uniques 1076; 2023-04-13 20:56:16,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681419376339, max timestamp=1681419376345Num uniques 1035; 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681419376339, max timestamp=1681419376345Num uniques 1035; 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1681419376339, max timestamp=1681419376345Num uniques 1035; 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 20:56:16,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-04-13 20:56:16,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419376353, max timestamp=1681419376357Num uniques 1044; 2023-04-13 20:56:16,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-13 20:56:16,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419376353, max timestamp=1681419376357Num uniques 1044; 2023-04-13 20:56:16,360 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-13 20:56:16,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419376353, max timestamp=1681419376357Num uniques 1044; 2023-04-13 20:56:16,361 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-04-13 20:56:16,378 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681419376369, max timestamp=1681419376376Num uniques 1103; 2023-04-13 20:56:16,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681419376369, max timestamp=1681419376376Num uniques 1103; 2023-04-13 20:56:16,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-13 20:56:16,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-13 20:56:16,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=253, cellSize=11585, totalHeapSize=26552, min timestamp=1681419376369, max timestamp=1681419376376Num uniques 1103; 2023-04-13 20:56:16,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=11585, suffix heap size=142680,new segment heap size=26552 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=253 2023-04-13 20:56:16,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419376390, max timestamp=1681419376395Num uniques 1112; 2023-04-13 20:56:16,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419376390, max timestamp=1681419376395Num uniques 1112; 2023-04-13 20:56:16,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-13 20:56:16,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-13 20:56:16,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419376390, max timestamp=1681419376395Num uniques 1112; 2023-04-13 20:56:16,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50922, new segment data size=7412, suffix heap size=143616,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1112, new segment cells count=162 2023-04-13 20:56:16,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1681419376406, max timestamp=1681419376411Num uniques 1021; 2023-04-13 20:56:16,413 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-13 20:56:16,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1681419376406, max timestamp=1681419376411Num uniques 1021; 2023-04-13 20:56:16,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-04-13 20:56:16,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376406, max timestamp=1681419376411Num uniques 1021; 2023-04-13 20:56:16,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=6870, suffix heap size=134152,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=150 2023-04-13 20:56:16,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419376416, max timestamp=1681419376425Num uniques 1080; 2023-04-13 20:56:16,427 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=8240, suffix heap size=140288,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=180 2023-04-13 20:56:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419376416, max timestamp=1681419376425Num uniques 1030; 2023-04-13 20:56:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419376416, max timestamp=1681419376425Num uniques 1009; 2023-04-13 20:56:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8240, suffix heap size=132904,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=180 2023-04-13 20:56:16,428 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-04-13 20:56:16,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681419376435, max timestamp=1681419376440Num uniques 1039; 2023-04-13 20:56:16,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-13 20:56:16,442 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681419376435, max timestamp=1681419376440Num uniques 1039; 2023-04-13 20:56:16,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-13 20:56:16,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1681419376435, max timestamp=1681419376440Num uniques 1039; 2023-04-13 20:56:16,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-13 20:56:16,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419376445, max timestamp=1681419376455Num uniques 1048; 2023-04-13 20:56:16,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:16,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419376445, max timestamp=1681419376455Num uniques 1048; 2023-04-13 20:56:16,458 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:16,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419376445, max timestamp=1681419376455Num uniques 1048; 2023-04-13 20:56:16,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:16,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681419376465, max timestamp=1681419376470Num uniques 1107; 2023-04-13 20:56:16,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1681419376465, max timestamp=1681419376470Num uniques 1107; 2023-04-13 20:56:16,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-04-13 20:56:16,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-04-13 20:56:16,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376465, max timestamp=1681419376470Num uniques 1107; 2023-04-13 20:56:16,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=6870, suffix heap size=143096,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=150 2023-04-13 20:56:16,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681419376484, max timestamp=1681419376487Num uniques 1009; 2023-04-13 20:56:16,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=7596, suffix heap size=132904,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=166 2023-04-13 20:56:16,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7591, totalHeapSize=17504, min timestamp=1681419376484, max timestamp=1681419376487Num uniques 1116; 2023-04-13 20:56:16,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51101, new segment data size=7591, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-04-13 20:56:16,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681419376484, max timestamp=1681419376487Num uniques 1116; 2023-04-13 20:56:16,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=7596, suffix heap size=144032,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=166 2023-04-13 20:56:16,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681419376532, max timestamp=1681419376536Num uniques 1025; 2023-04-13 20:56:16,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681419376532, max timestamp=1681419376536Num uniques 1025; 2023-04-13 20:56:16,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-13 20:56:16,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-13 20:56:16,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1681419376532, max timestamp=1681419376536Num uniques 1025; 2023-04-13 20:56:16,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=8010, suffix heap size=134568,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=175 2023-04-13 20:56:16,553 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304135c4a4ceef68642b28af761fd69a47c2d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c4a4ceef68642b28af761fd69a47c2d 2023-04-13 20:56:16,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8424, totalHeapSize=19376, min timestamp=1681419376544, max timestamp=1681419376550Num uniques 1034; 2023-04-13 20:56:16,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=8424, suffix heap size=135504,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=184 2023-04-13 20:56:16,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8414, totalHeapSize=19376, min timestamp=1681419376544, max timestamp=1681419376550Num uniques 1134; 2023-04-13 20:56:16,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51924, new segment data size=8414, suffix heap size=145904,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=184 2023-04-13 20:56:16,553 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=184, cellSize=8422, totalHeapSize=19376, min timestamp=1681419376544, max timestamp=1681419376550Num uniques 1034; 2023-04-13 20:56:16,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47352, new segment data size=8422, suffix heap size=135504,new segment heap size=19376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=184 2023-04-13 20:56:16,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1681419376556, max timestamp=1681419376564Num uniques 1043; 2023-04-13 20:56:16,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=13418, suffix heap size=136440,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=293 2023-04-13 20:56:16,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1681419376556, max timestamp=1681419376564Num uniques 1043; 2023-04-13 20:56:16,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=8838, suffix heap size=136440,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=193 2023-04-13 20:56:16,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376556, max timestamp=1681419376564Num uniques 1043; 2023-04-13 20:56:16,571 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47768, new segment data size=6870, suffix heap size=136440,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1043, new segment cells count=150 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1681419376577, max timestamp=1681419376581Num uniques 1009; 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=11540, suffix heap size=132904,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=252 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1681419376577, max timestamp=1681419376581Num uniques 1152; 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=252, cellSize=11540, totalHeapSize=26448, min timestamp=1681419376577, max timestamp=1681419376581Num uniques 1052; 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=11540, suffix heap size=147776,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=252 2023-04-13 20:56:16,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=11540, suffix heap size=137376,new segment heap size=26448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=252 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681419376593, max timestamp=1681419376595Num uniques 1111; 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681419376593, max timestamp=1681419376595Num uniques 1111; 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=7366, suffix heap size=143512,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=161 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=7366, suffix heap size=143512,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=161 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1681419376593, max timestamp=1681419376595Num uniques 1111; 2023-04-13 20:56:16,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50876, new segment data size=7366, suffix heap size=143512,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1111, new segment cells count=161 2023-04-13 20:56:16,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1681419376603, max timestamp=1681419376632Num uniques 1020; 2023-04-13 20:56:16,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 20:56:16,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1681419376603, max timestamp=1681419376632Num uniques 1020; 2023-04-13 20:56:16,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 20:56:16,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1681419376603, max timestamp=1681419376632Num uniques 1020; 2023-04-13 20:56:16,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-04-13 20:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681419376643, max timestamp=1681419376646Num uniques 1029; 2023-04-13 20:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376643, max timestamp=1681419376646Num uniques 1029; 2023-04-13 20:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-04-13 20:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=6870, suffix heap size=134984,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=150 2023-04-13 20:56:16,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376643, max timestamp=1681419376646Num uniques 1029; 2023-04-13 20:56:16,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=6870, suffix heap size=134984,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=150 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681419376656, max timestamp=1681419376660Num uniques 1038; 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681419376656, max timestamp=1681419376660Num uniques 1009; 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681419376656, max timestamp=1681419376660Num uniques 1009; 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10898, suffix heap size=132904,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=238 2023-04-13 20:56:16,663 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10898, suffix heap size=132904,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=238 2023-04-13 20:56:16,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681419376664, max timestamp=1681419376673Num uniques 1097; 2023-04-13 20:56:16,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-13 20:56:16,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681419376664, max timestamp=1681419376673Num uniques 1097; 2023-04-13 20:56:16,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-13 20:56:16,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1681419376664, max timestamp=1681419376673Num uniques 1097; 2023-04-13 20:56:16,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9426, totalHeapSize=21664, min timestamp=1681419376707, max timestamp=1681419376710Num uniques 1106; 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50646, new segment data size=9426, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9426, totalHeapSize=21664, min timestamp=1681419376707, max timestamp=1681419376710Num uniques 1106; 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=206, cellSize=9426, totalHeapSize=21664, min timestamp=1681419376707, max timestamp=1681419376710Num uniques 1106; 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50646, new segment data size=9426, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-13 20:56:16,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50646, new segment data size=9426, suffix heap size=142992,new segment heap size=21664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=206 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681419376720, max timestamp=1681419376724Num uniques 1065; 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681419376720, max timestamp=1681419376724Num uniques 1065; 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48770, new segment data size=7550, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7545, totalHeapSize=17400, min timestamp=1681419376720, max timestamp=1681419376724Num uniques 1065; 2023-04-13 20:56:16,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48765, new segment data size=7545, suffix heap size=138728,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1065, new segment cells count=165 2023-04-13 20:56:16,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681419376734, max timestamp=1681419376738Num uniques 1024; 2023-04-13 20:56:16,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-13 20:56:16,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681419376734, max timestamp=1681419376738Num uniques 1024; 2023-04-13 20:56:16,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-13 20:56:16,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=224, cellSize=10254, totalHeapSize=23536, min timestamp=1681419376734, max timestamp=1681419376738Num uniques 1024; 2023-04-13 20:56:16,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=10254, suffix heap size=134464,new segment heap size=23536 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=224 2023-04-13 20:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681419376747, max timestamp=1681419376755Num uniques 1083; 2023-04-13 20:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 20:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681419376747, max timestamp=1681419376755Num uniques 1083; 2023-04-13 20:56:16,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1681419376747, max timestamp=1681419376755Num uniques 1083; 2023-04-13 20:56:16,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 20:56:16,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49598, new segment data size=8378, suffix heap size=140600,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1083, new segment cells count=183 2023-04-13 20:56:16,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11072, totalHeapSize=25408, min timestamp=1681419376766, max timestamp=1681419376770Num uniques 1042; 2023-04-13 20:56:16,772 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47712, new segment data size=11072, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 20:56:16,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11072, totalHeapSize=25408, min timestamp=1681419376766, max timestamp=1681419376770Num uniques 1042; 2023-04-13 20:56:16,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=242, cellSize=11072, totalHeapSize=25408, min timestamp=1681419376766, max timestamp=1681419376770Num uniques 1042; 2023-04-13 20:56:16,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47712, new segment data size=11072, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 20:56:16,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47712, new segment data size=11072, suffix heap size=136336,new segment heap size=25408 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=242 2023-04-13 20:56:16,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681419376781, max timestamp=1681419376785Num uniques 1101; 2023-04-13 20:56:16,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 20:56:16,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681419376781, max timestamp=1681419376785Num uniques 1101; 2023-04-13 20:56:16,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1681419376781, max timestamp=1681419376785Num uniques 1101; 2023-04-13 20:56:16,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 20:56:16,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50425, new segment data size=11495, suffix heap size=142472,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1101, new segment cells count=251 2023-04-13 20:56:16,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1681419376793, max timestamp=1681419376800Num uniques 1110; 2023-04-13 20:56:16,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-13 20:56:16,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1681419376793, max timestamp=1681419376800Num uniques 1110; 2023-04-13 20:56:16,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=260, cellSize=11900, totalHeapSize=27280, min timestamp=1681419376793, max timestamp=1681419376800Num uniques 1110; 2023-04-13 20:56:16,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-13 20:56:16,802 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50830, new segment data size=11900, suffix heap size=143408,new segment heap size=27280 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1110, new segment cells count=260 2023-04-13 20:56:16,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419376809, max timestamp=1681419376816Num uniques 1119; 2023-04-13 20:56:16,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51236, new segment data size=7230, suffix heap size=144344,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=158 2023-04-13 20:56:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7725, totalHeapSize=17816, min timestamp=1681419376809, max timestamp=1681419376816Num uniques 1119; 2023-04-13 20:56:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51235, new segment data size=7725, suffix heap size=144344,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=169 2023-04-13 20:56:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=169, cellSize=7732, totalHeapSize=17816, min timestamp=1681419376809, max timestamp=1681419376816Num uniques 1119; 2023-04-13 20:56:16,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51242, new segment data size=7732, suffix heap size=144344,new segment heap size=17816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1119, new segment cells count=169 2023-04-13 20:56:16,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681419376821, max timestamp=1681419376833Num uniques 1028; 2023-04-13 20:56:16,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=10438, suffix heap size=134880,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=228 2023-04-13 20:56:16,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681419376821, max timestamp=1681419376833Num uniques 1017; 2023-04-13 20:56:16,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1681419376821, max timestamp=1681419376833Num uniques 1028; 2023-04-13 20:56:16,835 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46582, new segment data size=10438, suffix heap size=133736,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=228 2023-04-13 20:56:16,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47078, new segment data size=10438, suffix heap size=134880,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1028, new segment cells count=228 2023-04-13 20:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681419376849, max timestamp=1681419376854Num uniques 1087; 2023-04-13 20:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=8562, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-04-13 20:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1681419376849, max timestamp=1681419376854Num uniques 1087; 2023-04-13 20:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=8562, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-04-13 20:56:16,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376849, max timestamp=1681419376854Num uniques 1087; 2023-04-13 20:56:16,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=6870, suffix heap size=141016,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=150 2023-04-13 20:56:16,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681419376881, max timestamp=1681419376886Num uniques 1009; 2023-04-13 20:56:16,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11266, suffix heap size=132904,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=246 2023-04-13 20:56:16,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681419376881, max timestamp=1681419376886Num uniques 1046; 2023-04-13 20:56:16,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-13 20:56:16,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1681419376881, max timestamp=1681419376886Num uniques 1046; 2023-04-13 20:56:16,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-04-13 20:56:16,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1681419376900, max timestamp=1681419376905Num uniques 1105; 2023-04-13 20:56:16,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-04-13 20:56:16,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1681419376900, max timestamp=1681419376905Num uniques 1105; 2023-04-13 20:56:16,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1681419376900, max timestamp=1681419376905Num uniques 1105; 2023-04-13 20:56:16,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-04-13 20:56:16,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-04-13 20:56:16,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681419376916, max timestamp=1681419376919Num uniques 1014; 2023-04-13 20:56:16,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=7504, suffix heap size=133424,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=164 2023-04-13 20:56:16,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681419376916, max timestamp=1681419376919Num uniques 1014; 2023-04-13 20:56:16,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=7504, suffix heap size=133424,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=164 2023-04-13 20:56:16,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1681419376916, max timestamp=1681419376919Num uniques 1014; 2023-04-13 20:56:16,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=7504, suffix heap size=133424,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=164 2023-04-13 20:56:16,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7913, totalHeapSize=18232, min timestamp=1681419376927, max timestamp=1681419376940Num uniques 1023; 2023-04-13 20:56:16,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46843, new segment data size=7913, suffix heap size=134360,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=173 2023-04-13 20:56:16,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7915, totalHeapSize=18232, min timestamp=1681419376927, max timestamp=1681419376940Num uniques 1023; 2023-04-13 20:56:16,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46845, new segment data size=7915, suffix heap size=134360,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=173 2023-04-13 20:56:16,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=173, cellSize=7914, totalHeapSize=18232, min timestamp=1681419376927, max timestamp=1681419376940Num uniques 1023; 2023-04-13 20:56:16,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46844, new segment data size=7914, suffix heap size=134360,new segment heap size=18232 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=173 2023-04-13 20:56:16,961 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/e9c8cfe753ef4a9bba9cd3c391f237a8 2023-04-13 20:56:16,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1681419376983, max timestamp=1681419376988Num uniques 1032; 2023-04-13 20:56:16,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=10622, suffix heap size=135296,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=232 2023-04-13 20:56:16,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1681419376983, max timestamp=1681419376988Num uniques 1032; 2023-04-13 20:56:16,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=10622, suffix heap size=135296,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=232 2023-04-13 20:56:16,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419376985, max timestamp=1681419376988Num uniques 1032; 2023-04-13 20:56:16,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47262, new segment data size=6870, suffix heap size=135296,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1032, new segment cells count=150 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=291, cellSize=13326, totalHeapSize=30504, min timestamp=1681419376997, max timestamp=1681419377004Num uniques 1091; 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49966, new segment data size=13326, suffix heap size=141432,new segment heap size=30504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1091, new segment cells count=291 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=291, cellSize=13326, totalHeapSize=30504, min timestamp=1681419376997, max timestamp=1681419377004Num uniques 1091; 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=291, cellSize=13326, totalHeapSize=30504, min timestamp=1681419376997, max timestamp=1681419377004Num uniques 1009; 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=13326, suffix heap size=132904,new segment heap size=30504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=291 2023-04-13 20:56:17,007 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49966, new segment data size=13326, suffix heap size=141432,new segment heap size=30504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1091, new segment cells count=291 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681419377017, max timestamp=1681419377022Num uniques 1150; 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681419377017, max timestamp=1681419377022Num uniques 1150; 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52670, new segment data size=11450, suffix heap size=147568,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1150, new segment cells count=250 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=250, cellSize=11450, totalHeapSize=26240, min timestamp=1681419377017, max timestamp=1681419377022Num uniques 1150; 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=52670, new segment data size=11450, suffix heap size=147568,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1150, new segment cells count=250 2023-04-13 20:56:17,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52670, new segment data size=11450, suffix heap size=147568,new segment heap size=26240 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1150, new segment cells count=250 2023-04-13 20:56:17,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419377034, max timestamp=1681419377038Num uniques 1109; 2023-04-13 20:56:17,040 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=9565, suffix heap size=143304,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=209 2023-04-13 20:56:17,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419377034, max timestamp=1681419377038Num uniques 1109; 2023-04-13 20:56:17,041 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=9565, suffix heap size=143304,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=209 2023-04-13 20:56:17,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377035, max timestamp=1681419377038Num uniques 1109; 2023-04-13 20:56:17,042 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50785, new segment data size=6870, suffix heap size=143304,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1109, new segment cells count=150 2023-04-13 20:56:17,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419377048, max timestamp=1681419377051Num uniques 1009; 2023-04-13 20:56:17,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-13 20:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419377048, max timestamp=1681419377051Num uniques 1068; 2023-04-13 20:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419377048, max timestamp=1681419377051Num uniques 1068; 2023-04-13 20:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:17,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1681419377059, max timestamp=1681419377063Num uniques 1027; 2023-04-13 20:56:17,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-13 20:56:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377059, max timestamp=1681419377063Num uniques 1027; 2023-04-13 20:56:17,068 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-13 20:56:17,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377059, max timestamp=1681419377063Num uniques 1027; 2023-04-13 20:56:17,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-13 20:56:17,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419377079, max timestamp=1681419377085Num uniques 1086; 2023-04-13 20:56:17,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-13 20:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1681419377079, max timestamp=1681419377085Num uniques 1009; 2023-04-13 20:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-13 20:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377079, max timestamp=1681419377085Num uniques 1009; 2023-04-13 20:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-13 20:56:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419377091, max timestamp=1681419377127Num uniques 1045; 2023-04-13 20:56:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 20:56:17,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419377091, max timestamp=1681419377127Num uniques 1045; 2023-04-13 20:56:17,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-13 20:56:17,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1681419377091, max timestamp=1681419377127Num uniques 1009; 2023-04-13 20:56:17,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-04-13 20:56:17,166 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419377139, max timestamp=1681419377164Num uniques 1104; 2023-04-13 20:56:17,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-13 20:56:17,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419377139, max timestamp=1681419377164Num uniques 1104; 2023-04-13 20:56:17,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-13 20:56:17,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419377139, max timestamp=1681419377164Num uniques 1104; 2023-04-13 20:56:17,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-13 20:56:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419377169, max timestamp=1681419377182Num uniques 1013; 2023-04-13 20:56:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:56:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419377169, max timestamp=1681419377182Num uniques 1013; 2023-04-13 20:56:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:56:17,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1681419377169, max timestamp=1681419377182Num uniques 1013; 2023-04-13 20:56:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-04-13 20:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419377193, max timestamp=1681419377198Num uniques 1072; 2023-04-13 20:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419377193, max timestamp=1681419377198Num uniques 1072; 2023-04-13 20:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1681419377193, max timestamp=1681419377198Num uniques 1072; 2023-04-13 20:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419377209, max timestamp=1681419377218Num uniques 1081; 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1681419377209, max timestamp=1681419377218Num uniques 1081; 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377214, max timestamp=1681419377218Num uniques 1081; 2023-04-13 20:56:17,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=6870, suffix heap size=140392,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=150 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681419377229, max timestamp=1681419377234Num uniques 1090; 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681419377229, max timestamp=1681419377234Num uniques 1090; 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1681419377229, max timestamp=1681419377234Num uniques 1009; 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15570, suffix heap size=132904,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=340 2023-04-13 20:56:17,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=15570, suffix heap size=141328,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=340 2023-04-13 20:56:17,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1681419377244, max timestamp=1681419377253Num uniques 1199; 2023-04-13 20:56:17,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=9114, suffix heap size=152664,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=199 2023-04-13 20:56:17,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377244, max timestamp=1681419377253Num uniques 1199; 2023-04-13 20:56:17,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=6870, suffix heap size=152664,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=150 2023-04-13 20:56:17,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377244, max timestamp=1681419377253Num uniques 1199; 2023-04-13 20:56:17,261 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=54914, new segment data size=6870, suffix heap size=152664,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1199, new segment cells count=150 2023-04-13 20:56:17,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419377263, max timestamp=1681419377268Num uniques 1058; 2023-04-13 20:56:17,270 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419377263, max timestamp=1681419377268Num uniques 1009; 2023-04-13 20:56:17,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419377263, max timestamp=1681419377268Num uniques 1009; 2023-04-13 20:56:17,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-13 20:56:17,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7230, suffix heap size=132904,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=158 2023-04-13 20:56:17,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=7230, suffix heap size=132904,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=158 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681419377276, max timestamp=1681419377280Num uniques 1017; 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681419377276, max timestamp=1681419377280Num uniques 1017; 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1681419377276, max timestamp=1681419377280Num uniques 1017; 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-13 20:56:17,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-04-13 20:56:17,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681419377290, max timestamp=1681419377317Num uniques 1126; 2023-04-13 20:56:17,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-13 20:56:17,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377313, max timestamp=1681419377317Num uniques 1126; 2023-04-13 20:56:17,341 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=6870, suffix heap size=145072,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=150 2023-04-13 20:56:17,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1681419377290, max timestamp=1681419377317Num uniques 1126; 2023-04-13 20:56:17,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681419377348, max timestamp=1681419377352Num uniques 1135; 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681419377348, max timestamp=1681419377352Num uniques 1009; 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=15340, suffix heap size=132904,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=335 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=335, cellSize=15340, totalHeapSize=35080, min timestamp=1681419377348, max timestamp=1681419377352Num uniques 1135; 2023-04-13 20:56:17,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=15340, suffix heap size=146008,new segment heap size=35080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=335 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419377361, max timestamp=1681419377365Num uniques 1194; 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419377361, max timestamp=1681419377365Num uniques 1194; 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1681419377361, max timestamp=1681419377365Num uniques 1194; 2023-04-13 20:56:17,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=54684, new segment data size=11174, suffix heap size=152144,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1194, new segment cells count=244 2023-04-13 20:56:17,372 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/0dccd0fc91714de0a0202d1f6f2e1970 2023-04-13 20:56:17,381 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681419377380, max timestamp=1681419377394Num uniques 1103; 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681419377380, max timestamp=1681419377394Num uniques 1103; 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1681419377380, max timestamp=1681419377394Num uniques 1103; 2023-04-13 20:56:17,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-13 20:56:17,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419377410, max timestamp=1681419377441Num uniques 1062; 2023-04-13 20:56:17,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-13 20:56:17,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=262, cellSize=11992, totalHeapSize=27488, min timestamp=1681419377409, max timestamp=1681419377441Num uniques 1062; 2023-04-13 20:56:17,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=11992, suffix heap size=138416,new segment heap size=27488 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=262 2023-04-13 20:56:17,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1681419377410, max timestamp=1681419377441Num uniques 1062; 2023-04-13 20:56:17,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12398, totalHeapSize=28424, min timestamp=1681419377448, max timestamp=1681419377457Num uniques 1121; 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51328, new segment data size=12398, suffix heap size=144552,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1121, new segment cells count=271 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12396, totalHeapSize=28424, min timestamp=1681419377448, max timestamp=1681419377457Num uniques 1021; 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=271, cellSize=12406, totalHeapSize=28424, min timestamp=1681419377448, max timestamp=1681419377457Num uniques 1021; 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=12406, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-04-13 20:56:17,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46746, new segment data size=12396, suffix heap size=134152,new segment heap size=28424 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=271 2023-04-13 20:56:17,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681419377464, max timestamp=1681419377474Num uniques 1130; 2023-04-13 20:56:17,476 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-13 20:56:17,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1681419377464, max timestamp=1681419377474Num uniques 1130; 2023-04-13 20:56:17,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1681419377464, max timestamp=1681419377474Num uniques 1130; 2023-04-13 20:56:17,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=10530, suffix heap size=145488,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=230 2023-04-13 20:56:17,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51750, new segment data size=8240, suffix heap size=145488,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1130, new segment cells count=180 2023-04-13 20:56:17,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8644, totalHeapSize=19896, min timestamp=1681419377487, max timestamp=1681419377489Num uniques 1089; 2023-04-13 20:56:17,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49864, new segment data size=8644, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-13 20:56:17,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8645, totalHeapSize=19896, min timestamp=1681419377487, max timestamp=1681419377489Num uniques 1089; 2023-04-13 20:56:17,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49865, new segment data size=8645, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-04-13 20:56:17,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8644, totalHeapSize=19896, min timestamp=1681419377487, max timestamp=1681419377489Num uniques 1039; 2023-04-13 20:56:17,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47574, new segment data size=8644, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-04-13 20:56:17,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419377496, max timestamp=1681419377502Num uniques 1048; 2023-04-13 20:56:17,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:17,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419377496, max timestamp=1681419377502Num uniques 1048; 2023-04-13 20:56:17,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:17,505 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1681419377496, max timestamp=1681419377502Num uniques 1048; 2023-04-13 20:56:17,506 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681419377513, max timestamp=1681419377516Num uniques 1107; 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681419377513, max timestamp=1681419377516Num uniques 1107; 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1681419377513, max timestamp=1681419377516Num uniques 1107; 2023-04-13 20:56:17,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-04-13 20:56:17,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681419377526, max timestamp=1681419377553Num uniques 1066; 2023-04-13 20:56:17,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 20:56:17,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681419377526, max timestamp=1681419377553Num uniques 1066; 2023-04-13 20:56:17,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 20:56:17,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=166, cellSize=7596, totalHeapSize=17504, min timestamp=1681419377526, max timestamp=1681419377553Num uniques 1066; 2023-04-13 20:56:17,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=7596, suffix heap size=138832,new segment heap size=17504 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=166 2023-04-13 20:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681419377562, max timestamp=1681419377567Num uniques 1025; 2023-04-13 20:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-13 20:56:17,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681419377562, max timestamp=1681419377567Num uniques 1025; 2023-04-13 20:56:17,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-13 20:56:17,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=275, cellSize=12590, totalHeapSize=28840, min timestamp=1681419377562, max timestamp=1681419377567Num uniques 1025; 2023-04-13 20:56:17,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46940, new segment data size=12590, suffix heap size=134568,new segment heap size=28840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1025, new segment cells count=275 2023-04-13 20:56:17,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681419377576, max timestamp=1681419377580Num uniques 1134; 2023-04-13 20:56:17,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=10714, suffix heap size=145904,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=234 2023-04-13 20:56:17,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681419377576, max timestamp=1681419377580Num uniques 1134; 2023-04-13 20:56:17,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=10714, suffix heap size=145904,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=234 2023-04-13 20:56:17,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1681419377576, max timestamp=1681419377580Num uniques 1134; 2023-04-13 20:56:17,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51934, new segment data size=10714, suffix heap size=145904,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1134, new segment cells count=234 2023-04-13 20:56:17,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681419377590, max timestamp=1681419377594Num uniques 1093; 2023-04-13 20:56:17,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 20:56:17,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681419377590, max timestamp=1681419377594Num uniques 1093; 2023-04-13 20:56:17,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 20:56:17,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=243, cellSize=11128, totalHeapSize=25512, min timestamp=1681419377590, max timestamp=1681419377594Num uniques 1093; 2023-04-13 20:56:17,597 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=11128, suffix heap size=141640,new segment heap size=25512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=243 2023-04-13 20:56:17,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681419377602, max timestamp=1681419377611Num uniques 1102; 2023-04-13 20:56:17,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-13 20:56:17,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681419377602, max timestamp=1681419377611Num uniques 1102; 2023-04-13 20:56:17,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-13 20:56:17,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=202, cellSize=9250, totalHeapSize=21248, min timestamp=1681419377602, max timestamp=1681419377611Num uniques 1102; 2023-04-13 20:56:17,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50470, new segment data size=9250, suffix heap size=142576,new segment heap size=21248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1102, new segment cells count=202 2023-04-13 20:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1681419377619, max timestamp=1681419377624Num uniques 1061; 2023-04-13 20:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1681419377619, max timestamp=1681419377624Num uniques 1061; 2023-04-13 20:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=11946, suffix heap size=138312,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=261 2023-04-13 20:56:17,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=261, cellSize=11946, totalHeapSize=27384, min timestamp=1681419377619, max timestamp=1681419377624Num uniques 1061; 2023-04-13 20:56:17,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=11946, suffix heap size=138312,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=261 2023-04-13 20:56:17,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48586, new segment data size=11946, suffix heap size=138312,new segment heap size=27384 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1061, new segment cells count=261 2023-04-13 20:56:17,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681419377641, max timestamp=1681419377648Num uniques 1120; 2023-04-13 20:56:17,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=10070, suffix heap size=144448,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=220 2023-04-13 20:56:17,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1681419377641, max timestamp=1681419377648Num uniques 1120; 2023-04-13 20:56:17,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=10070, suffix heap size=144448,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=220 2023-04-13 20:56:17,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1681419377644, max timestamp=1681419377648Num uniques 1120; 2023-04-13 20:56:17,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51290, new segment data size=6870, suffix heap size=144448,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1120, new segment cells count=150 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681419377705, max timestamp=1681419377707Num uniques 1079; 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681419377705, max timestamp=1681419377707Num uniques 1009; 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1681419377705, max timestamp=1681419377707Num uniques 1079; 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=8194, suffix heap size=140184,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=179 2023-04-13 20:56:17,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8194, suffix heap size=132904,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=179 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1681419377720, max timestamp=1681419377724Num uniques 1038; 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=288, cellSize=13188, totalHeapSize=30192, min timestamp=1681419377720, max timestamp=1681419377724Num uniques 1038; 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=13188, suffix heap size=135920,new segment heap size=30192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=288 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1681419377720, max timestamp=1681419377724Num uniques 1038; 2023-04-13 20:56:17,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-04-13 20:56:17,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681419377732, max timestamp=1681419377738Num uniques 1097; 2023-04-13 20:56:17,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-04-13 20:56:17,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681419377732, max timestamp=1681419377738Num uniques 1147; 2023-04-13 20:56:17,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52532, new segment data size=9022, suffix heap size=147256,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1147, new segment cells count=197 2023-04-13 20:56:17,742 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1681419377732, max timestamp=1681419377738Num uniques 1047; 2023-04-13 20:56:17,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=9022, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681419377751, max timestamp=1681419377757Num uniques 1056; 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681419377751, max timestamp=1681419377757Num uniques 1056; 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1681419377751, max timestamp=1681419377757Num uniques 1056; 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-13 20:56:17,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7545, totalHeapSize=17400, min timestamp=1681419377765, max timestamp=1681419377771Num uniques 1015; 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7550, totalHeapSize=17400, min timestamp=1681419377765, max timestamp=1681419377771Num uniques 1015; 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7546, totalHeapSize=17400, min timestamp=1681419377765, max timestamp=1681419377771Num uniques 1015; 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46475, new segment data size=7545, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46476, new segment data size=7546, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-13 20:56:17,774 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46480, new segment data size=7550, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-04-13 20:56:17,794 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/e3032d7db71743bea4020829595b2fd8 2023-04-13 20:56:17,798 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/e9c8cfe753ef4a9bba9cd3c391f237a8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8 2023-04-13 20:56:17,801 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8, entries=50, sequenceid=8, filesize=11.6 K 2023-04-13 20:56:17,801 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/0dccd0fc91714de0a0202d1f6f2e1970 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970 2023-04-13 20:56:17,805 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970, entries=50, sequenceid=8, filesize=7.0 K 2023-04-13 20:56:17,805 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/e3032d7db71743bea4020829595b2fd8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8 2023-04-13 20:56:17,809 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8, entries=50, sequenceid=8, filesize=7.0 K 2023-04-13 20:56:17,819 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=87.22 KB/89310 for b19b9c1a7b3fa24b26a81e15015b427d in 1707ms, sequenceid=8, compaction requested=false 2023-04-13 20:56:17,819 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-13 20:56:17,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:17,820 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:17,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:17,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=114.05 KB heapSize=324.66 KB 2023-04-13 20:56:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:17,820 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:17,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:17,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:17,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:17,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:17,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:17,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:17,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,822 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,822 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:17,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39efabe4[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:17,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:17,824 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,827 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:17,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:17,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:17,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,827 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:17,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:17,829 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:17,829 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:17,829 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:17,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:17,831 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:17,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:17,831 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:17,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:17,832 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:17,832 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d550291bfd5548ca801ee06ce129ee1a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d550291bfd5548ca801ee06ce129ee1a 2023-04-13 20:56:17,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:17,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,834 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,835 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419437832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419437833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419437835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:17,839 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:17,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:17,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,839 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:17,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419437836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:17,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:17,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:17,839 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:17,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:17,840 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419437840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,940 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:17,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:17,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419437939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:17,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419437940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419437940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419437940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:17,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:17,943 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:17,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:17,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:17,943 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:17,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:17,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419437943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,944 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,944 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,945 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:17,945 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:17,945 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,945 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:17,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:17,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:17,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:17,946 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:17,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:17,946 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:17,947 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:17,947 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:17,947 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,948 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:17,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f44f5b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:17,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:17,950 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,951 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:17,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:17,954 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:17,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:17,954 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:17,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:17,956 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:17,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:17,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:17,956 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:17,956 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:17,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:17,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:17,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:17,958 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:17,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:17,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:17,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:17,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:17,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:17,962 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:17,962 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:17,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:17,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:17,963 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:17,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:18,062 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:18,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:18,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:18,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:18,066 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:18,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:18,066 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:18,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,067 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,068 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:18,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:18,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,068 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:18,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:18,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:18,070 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:18,070 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:18,070 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:18,070 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:18,074 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,075 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:18,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ded0ab7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:18,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:18,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,077 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,078 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:18,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:18,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:18,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:18,082 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:18,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:18,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:18,082 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:18,082 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:18,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,085 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:18,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:18,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:18,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:18,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:18,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:18,089 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:18,089 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:18,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:18,088 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:18,089 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419438143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419438143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419438144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419438144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419438144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,189 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:18,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:18,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:18,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:18,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:18,192 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:18,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:18,192 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:18,193 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,193 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:18,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:18,194 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,195 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,195 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:18,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:18,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:18,196 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:18,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:18,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:18,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:18,197 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:18,197 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:18,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,198 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:18,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497a737e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:18,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:18,201 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,202 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:18,202 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:18,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:18,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,205 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:18,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:18,208 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:18,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:18,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:18,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:18,208 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:18,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:18,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:18,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:18,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:18,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,212 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:18,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:18,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,216 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:18,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:18,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:18,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:18,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:18,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:18,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:18,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:18,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:18,217 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,217 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:18,217 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:18,237 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1485, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/9d40a97a82904e6c9bc79bc1fc38ddf8 2023-04-13 20:56:18,249 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1485 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/1d061409580446fba5e69aa9642a5819 2023-04-13 20:56:18,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1485 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/56f141414b934f3db87340410b1aed3d 2023-04-13 20:56:18,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/9d40a97a82904e6c9bc79bc1fc38ddf8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8 2023-04-13 20:56:18,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8, entries=200, sequenceid=1485, filesize=32.3 K 2023-04-13 20:56:18,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/1d061409580446fba5e69aa9642a5819 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819 2023-04-13 20:56:18,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819, entries=150, sequenceid=1485, filesize=11.9 K 2023-04-13 20:56:18,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/56f141414b934f3db87340410b1aed3d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d 2023-04-13 20:56:18,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d, entries=150, sequenceid=1485, filesize=11.9 K 2023-04-13 20:56:18,285 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~324.61 KB/332400, currentSize=67.09 KB/68700 for b19b9c1a7b3fa24b26a81e15015b427d in 465ms, sequenceid=1485, compaction requested=false 2023-04-13 20:56:18,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:18,317 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:18,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:18,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:18,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:18,321 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:18,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:18,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:18,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,321 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,327 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:18,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:18,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:18,327 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,328 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:18,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:18,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:18,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:18,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:18,329 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:18,329 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:18,329 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:18,329 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-13 20:56:18,329 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:18,329 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:18,330 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:18,330 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:18,330 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:18,330 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:18,341 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304133d461a96a52d4cd3876bfcc8e54b90af to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133d461a96a52d4cd3876bfcc8e54b90af 2023-04-13 20:56:18,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:18,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419438462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419438464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419438464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,468 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419438465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419438468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419438567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,568 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419438568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419438568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419438569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419438571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,753 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1498, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/30a6b8c5b0b842528ba6c4490c460249 2023-04-13 20:56:18,765 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1498 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/78df554aad6c42f9b4b7feb577aeba08 2023-04-13 20:56:18,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419438769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419438770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419438771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419438772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:18,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419438773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419439072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419439072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419439073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419439075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,077 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419439076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,177 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1498 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/b2c0f68f60ac401d95831c18e97fe8c9 2023-04-13 20:56:19,181 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/30a6b8c5b0b842528ba6c4490c460249 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249 2023-04-13 20:56:19,184 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249, entries=150, sequenceid=1498, filesize=25.4 K 2023-04-13 20:56:19,185 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/78df554aad6c42f9b4b7feb577aeba08 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08 2023-04-13 20:56:19,188 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08, entries=150, sequenceid=1498, filesize=11.9 K 2023-04-13 20:56:19,189 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/b2c0f68f60ac401d95831c18e97fe8c9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9 2023-04-13 20:56:19,192 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9, entries=150, sequenceid=1498, filesize=11.9 K 2023-04-13 20:56:19,192 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for b19b9c1a7b3fa24b26a81e15015b427d in 863ms, sequenceid=1498, compaction requested=true 2023-04-13 20:56:19,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:19,192 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:19,192 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:19,193 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:19,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,194 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,195 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:19,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d0e413e[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:19,196 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:19,196 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,198 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:19,198 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:19,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,198 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:19,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:19,199 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,200 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,200 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:19,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:19,202 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:19,202 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:19,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:19,203 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:19,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,207 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,209 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:19,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,209 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:19,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:19,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:19,210 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:19,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,210 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:19,310 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:19,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:19,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:19,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:19,314 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:19,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:19,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:19,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:19,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,316 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:19,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:19,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:19,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,316 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:19,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:19,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:19,317 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:19,318 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:19,318 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,318 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:19,319 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,319 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:19,319 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:19,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:19,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419439584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419439585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419439586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419439587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419439587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419439687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419439689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419439690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419439690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419439690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,733 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041369247f18b29849c89a7c34de9234966a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041369247f18b29849c89a7c34de9234966a 2023-04-13 20:56:19,738 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1517, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/d8b15c89a4ef4025b68baedd5c1a1a36 2023-04-13 20:56:19,750 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1517 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/21f86875e73640e5b325364d9ab0fa1a 2023-04-13 20:56:19,760 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1517 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/577017a3bd9a42829e24a12abe05c0a5 2023-04-13 20:56:19,764 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/d8b15c89a4ef4025b68baedd5c1a1a36 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36 2023-04-13 20:56:19,768 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36, entries=150, sequenceid=1517, filesize=25.4 K 2023-04-13 20:56:19,769 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/21f86875e73640e5b325364d9ab0fa1a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a 2023-04-13 20:56:19,773 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a, entries=150, sequenceid=1517, filesize=11.9 K 2023-04-13 20:56:19,773 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/577017a3bd9a42829e24a12abe05c0a5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5 2023-04-13 20:56:19,777 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5, entries=150, sequenceid=1517, filesize=11.9 K 2023-04-13 20:56:19,777 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for b19b9c1a7b3fa24b26a81e15015b427d in 459ms, sequenceid=1517, compaction requested=true 2023-04-13 20:56:19,777 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:19,777 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:19,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:19,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:19,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:19,778 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:19,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,779 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,779 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:19,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60b4953[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:19,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:19,781 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,783 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:19,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:19,783 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:19,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,783 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:19,785 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,785 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:19,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:19,787 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:19,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:19,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:19,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:19,788 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:19,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,790 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,791 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,795 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:19,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:19,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,795 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,795 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:19,795 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:19,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:19,892 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:19,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:19,895 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:19,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:19,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:19,901 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:19,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:19,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:19,901 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:19,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,902 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,903 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,903 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:19,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:19,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:19,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,904 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:19,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:19,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:19,905 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419439904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419439904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419439907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419439908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:19,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419439908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:19,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:19,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:19,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:19,912 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:19,912 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:19,914 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,914 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:19,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33864397[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:19,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:19,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,917 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:19,917 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:19,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:19,919 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,919 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:19,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:19,922 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:19,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:19,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:19,922 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:19,922 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:19,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:19,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,930 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:19,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:19,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,931 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:19,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:19,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:19,934 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:19,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:19,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:19,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:19,934 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:19,934 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:19,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:19,935 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419440009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419440009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419440010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419440011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419440011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,035 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,038 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,038 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,039 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,039 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,040 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,041 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,041 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,041 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,041 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,042 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,042 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,043 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,043 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fd271a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,050 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,050 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,050 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,051 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,051 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,052 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,054 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,054 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,054 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,054 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,054 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,055 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,056 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,057 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,059 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,060 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,060 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:20,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,160 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,163 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,166 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,169 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,169 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,169 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,169 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,169 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,171 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@560584aa[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,175 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,179 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,184 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,185 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,188 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,188 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419440213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419440213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419440213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419440215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419440216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,289 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,292 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,293 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,295 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,297 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,297 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,297 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,298 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,298 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a181ee8[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,306 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,306 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,306 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,307 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,309 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d93ceb39405a4e228e195cb8cc47863d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d93ceb39405a4e228e195cb8cc47863d 2023-04-13 20:56:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,311 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,311 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,311 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,313 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,313 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,314 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,314 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,315 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,318 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,319 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,319 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,320 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,320 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,320 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:20,420 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,423 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,423 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,426 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,426 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,427 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,428 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,428 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,430 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,430 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b837ec7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,432 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,433 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,433 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,437 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,437 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,440 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,440 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,440 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,441 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,442 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,443 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,446 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,446 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,446 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,446 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419440516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419440517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419440518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419440518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:20,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419440520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,546 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,552 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,554 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,557 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,558 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,559 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,560 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,560 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,560 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,560 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,564 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b064d5a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,573 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,575 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,575 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,576 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,577 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,577 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,580 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,580 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,580 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,582 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,584 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:20,588 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,589 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,689 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,692 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,692 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,696 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,697 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,697 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,698 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,700 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,701 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,701 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,703 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,704 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@315cb044[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,705 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,708 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,708 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,708 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,709 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,709 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,712 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,712 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,714 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,715 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,718 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,718 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,718 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,718 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,719 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,719 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,719 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,729 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1532, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/c1f6908200c34ad38f006d8d4015fe6c 2023-04-13 20:56:20,819 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,822 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,822 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,823 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,823 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,825 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,825 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,826 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,827 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,827 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,827 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,827 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,827 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,828 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,829 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,829 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7907d9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,830 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,831 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,832 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,832 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,833 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,833 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,836 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,836 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,836 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,840 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,842 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,846 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,847 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,847 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,847 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,847 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:20,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,947 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:20,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:20,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:20,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:20,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:20,951 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:20,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:20,951 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:20,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,952 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,953 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,953 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:20,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:20,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:20,955 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:20,956 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,956 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:20,956 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:20,956 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:20,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,958 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:20,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b1c2b37[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:20,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:20,960 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,962 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,963 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,963 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:20,963 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:20,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,964 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:20,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:20,968 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:20,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:20,969 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:20,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,971 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,973 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:20,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:20,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:20,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:20,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:20,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:20,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:20,975 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:20,975 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:20,976 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:20,976 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:20,976 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:20,976 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:20,976 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:21,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419441022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419441022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:21,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419441022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:21,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419441024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:21,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419441028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,076 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,079 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,079 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,080 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,084 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,085 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,085 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:21,085 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,087 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1209c793[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,090 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,090 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,091 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,091 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,091 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,092 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,095 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,095 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,098 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,100 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,103 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,103 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,103 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,103 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:21,104 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,104 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,103 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,139 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1532 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/5aea6bdc13c1461e93aa66807286ea55 2023-04-13 20:56:21,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1532 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/4c6b3203e82442c9bf22909977e6b8b0 2023-04-13 20:56:21,151 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/c1f6908200c34ad38f006d8d4015fe6c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c 2023-04-13 20:56:21,154 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c, entries=150, sequenceid=1532, filesize=25.4 K 2023-04-13 20:56:21,154 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/5aea6bdc13c1461e93aa66807286ea55 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55 2023-04-13 20:56:21,158 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55, entries=150, sequenceid=1532, filesize=11.9 K 2023-04-13 20:56:21,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/4c6b3203e82442c9bf22909977e6b8b0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0 2023-04-13 20:56:21,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0, entries=150, sequenceid=1532, filesize=11.9 K 2023-04-13 20:56:21,163 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for b19b9c1a7b3fa24b26a81e15015b427d in 1271ms, sequenceid=1532, compaction requested=true 2023-04-13 20:56:21,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,163 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:21,163 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:21,163 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:21,163 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:21,163 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:21,164 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 123080 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:21,164 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55944 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:21,164 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:21,164 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:21,164 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,164 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,165 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=120.2 K 2023-04-13 20:56:21,165 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=54.6 K 2023-04-13 20:56:21,165 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e9c8cfe753ef4a9bba9cd3c391f237a8, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681419376106 2023-04-13 20:56:21,165 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0dccd0fc91714de0a0202d1f6f2e1970, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681419376106 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 9d40a97a82904e6c9bc79bc1fc38ddf8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1485, earliestPutTs=1681419377789 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1d061409580446fba5e69aa9642a5819, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1485, earliestPutTs=1681419377793 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 30a6b8c5b0b842528ba6c4490c460249, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1498, earliestPutTs=1681419377831 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 78df554aad6c42f9b4b7feb577aeba08, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1498, earliestPutTs=1681419377831 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d8b15c89a4ef4025b68baedd5c1a1a36, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1517, earliestPutTs=1681419378451 2023-04-13 20:56:21,166 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 21f86875e73640e5b325364d9ab0fa1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1517, earliestPutTs=1681419378451 2023-04-13 20:56:21,167 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting c1f6908200c34ad38f006d8d4015fe6c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1681419379581 2023-04-13 20:56:21,167 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5aea6bdc13c1461e93aa66807286ea55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1681419379581 2023-04-13 20:56:21,175 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#596 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:21,182 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#597 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:21,204 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,208 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,208 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,210 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,211 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,211 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,212 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:21,212 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:21,224 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f470096c3e584bb89c3e3e7dacd4292d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f470096c3e584bb89c3e3e7dacd4292d 2023-04-13 20:56:21,229 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1550, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/331723f764cb464e856f4b60839b4f55 2023-04-13 20:56:21,241 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1550 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/612461aab0ed48d385a41a3a1ce693bd 2023-04-13 20:56:21,248 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1550 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/8b2e379ba7af438db3851415d9644a72 2023-04-13 20:56:21,251 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/331723f764cb464e856f4b60839b4f55 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55 2023-04-13 20:56:21,254 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55, entries=150, sequenceid=1550, filesize=25.4 K 2023-04-13 20:56:21,254 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/612461aab0ed48d385a41a3a1ce693bd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd 2023-04-13 20:56:21,257 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd, entries=150, sequenceid=1550, filesize=11.9 K 2023-04-13 20:56:21,258 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/8b2e379ba7af438db3851415d9644a72 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72 2023-04-13 20:56:21,262 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72, entries=150, sequenceid=1550, filesize=11.9 K 2023-04-13 20:56:21,262 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for b19b9c1a7b3fa24b26a81e15015b427d in 51ms, sequenceid=1550, compaction requested=true 2023-04-13 20:56:21,263 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,263 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,263 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,264 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46039b06[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,266 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,269 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,269 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,273 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,273 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,276 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,277 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:21,283 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,283 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,283 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,383 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,387 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,387 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,388 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,388 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,389 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,389 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,391 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,391 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,393 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75771bd0[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,395 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,396 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,397 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,398 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,401 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,401 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,401 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,401 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,401 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,403 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,407 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,407 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,407 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,407 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,406 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,407 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,407 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,507 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,511 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,511 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,514 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,514 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,515 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,516 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,516 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,518 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f38c3da[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,521 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,522 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,522 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,523 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,525 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,525 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,525 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,527 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,527 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,528 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,529 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,533 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,533 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,533 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,534 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,590 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/077b6e6ff527442186b98ab06e8d9281 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/077b6e6ff527442186b98ab06e8d9281 2023-04-13 20:56:21,594 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into 077b6e6ff527442186b98ab06e8d9281(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:21,594 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,595 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=11, startTime=1681419381163; duration=0sec 2023-04-13 20:56:21,595 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:21,595 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-13 20:56:21,596 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 68130 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-13 20:56:21,597 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:21,597 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,597 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=66.5 K 2023-04-13 20:56:21,597 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e3032d7db71743bea4020829595b2fd8, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1681419376106 2023-04-13 20:56:21,597 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 56f141414b934f3db87340410b1aed3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1485, earliestPutTs=1681419377793 2023-04-13 20:56:21,598 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b2c0f68f60ac401d95831c18e97fe8c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1498, earliestPutTs=1681419377831 2023-04-13 20:56:21,598 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 577017a3bd9a42829e24a12abe05c0a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1517, earliestPutTs=1681419378451 2023-04-13 20:56:21,598 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4c6b3203e82442c9bf22909977e6b8b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1681419379581 2023-04-13 20:56:21,599 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8b2e379ba7af438db3851415d9644a72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1550, earliestPutTs=1681419379899 2023-04-13 20:56:21,607 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#601 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:21,614 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/6091ee476c9d45d79f7dded201a3ce12 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/6091ee476c9d45d79f7dded201a3ce12 2023-04-13 20:56:21,615 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/978b20e090104056990883adb90ab67d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/978b20e090104056990883adb90ab67d 2023-04-13 20:56:21,618 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into 6091ee476c9d45d79f7dded201a3ce12(size=25.6 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:21,618 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,618 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=11, startTime=1681419381163; duration=0sec 2023-04-13 20:56:21,618 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:21,621 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into 978b20e090104056990883adb90ab67d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:21,621 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,621 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=10, startTime=1681419381163; duration=0sec 2023-04-13 20:56:21,621 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:21,634 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,637 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,637 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,642 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,643 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,643 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,644 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,647 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,647 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d1a6f1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,649 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,656 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,656 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,660 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,660 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,660 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,662 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,662 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,662 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,662 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,666 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,666 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:21,784 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,788 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,789 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,793 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,794 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,795 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,795 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,795 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,796 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,798 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7950c226[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,800 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,802 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,802 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,802 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,804 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,804 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,807 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,807 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,807 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,807 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,807 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,809 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,811 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,813 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,814 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,814 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,815 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,815 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,915 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:21,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:21,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:21,918 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:21,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:21,918 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:21,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,919 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,920 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,920 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:21,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:21,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:21,922 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,922 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:21,922 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:21,922 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:21,922 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:21,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3434ef47[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:21,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:21,927 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,928 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:21,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,928 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:21,929 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:21,932 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:21,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:21,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:21,932 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:21,932 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:21,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:21,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,936 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:21,939 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:21,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:21,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:21,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:21,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:21,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:21,940 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:21,940 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,940 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:21,940 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:21,940 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:21,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:21,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:21,940 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,040 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:22,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:22,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:22,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:22,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:22,046 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:22,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:22,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:22,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,046 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,048 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,048 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:22,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:22,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:22,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:22,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:22,049 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,049 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:22,050 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-13 20:56:22,050 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:22,050 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,050 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:22,051 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,051 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:22,051 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:22,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:22,077 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413158a5b37ea454237875013419a0089ac to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413158a5b37ea454237875013419a0089ac 2023-04-13 20:56:22,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419382070, max timestamp=1681419382079Num uniques 859; 2023-04-13 20:56:22,081 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:22,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419382070, max timestamp=1681419382079Num uniques 859; 2023-04-13 20:56:22,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:22,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419382070, max timestamp=1681419382079Num uniques 859; 2023-04-13 20:56:22,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:22,096 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1564, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/0a551e37095143b79f01b230b4dab5d7 2023-04-13 20:56:22,108 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1564 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/7029a2a7790c4edcb8241f9194082153 2023-04-13 20:56:22,115 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1564 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/8ca75e670e444cfe924b9302f4618077 2023-04-13 20:56:22,118 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/0a551e37095143b79f01b230b4dab5d7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7 2023-04-13 20:56:22,120 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7, entries=150, sequenceid=1564, filesize=25.4 K 2023-04-13 20:56:22,121 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/7029a2a7790c4edcb8241f9194082153 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153 2023-04-13 20:56:22,123 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153, entries=100, sequenceid=1564, filesize=9.5 K 2023-04-13 20:56:22,124 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/8ca75e670e444cfe924b9302f4618077 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077 2023-04-13 20:56:22,126 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077, entries=100, sequenceid=1564, filesize=9.5 K 2023-04-13 20:56:22,135 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=87.22 KB/89310 for b19b9c1a7b3fa24b26a81e15015b427d in 85ms, sequenceid=1564, compaction requested=true 2023-04-13 20:56:22,135 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:22,135 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:22,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:22,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=107.34 KB heapSize=305.30 KB 2023-04-13 20:56:22,136 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:22,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:22,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:22,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:22,137 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:22,137 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,138 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,138 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:22,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dabdc08[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:22,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:22,140 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,146 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:22,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,146 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:22,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,148 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,149 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:22,152 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:22,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:22,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:22,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:22,152 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:22,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419442149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419442149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,154 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419442150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,154 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,154 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,155 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419442151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419442152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,156 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,159 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:22,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,159 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:22,159 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:22,159 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:22,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419442255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419442255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419442255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419442256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419442257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,259 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:22,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:22,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:22,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:22,263 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:22,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:22,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:22,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,263 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,264 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,265 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,265 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,265 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:22,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:22,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:22,266 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:22,267 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,267 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:22,267 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:22,267 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:22,267 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:22,270 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23630562[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:22,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:22,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,275 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:22,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:22,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,275 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,277 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,277 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:22,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:22,280 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:22,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:22,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:22,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:22,280 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:22,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,284 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:22,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:22,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,288 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:22,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,289 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:22,289 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:22,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,389 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:22,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:22,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:22,392 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:22,392 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,392 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:22,392 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:22,393 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,393 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,394 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,394 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:22,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:22,396 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,396 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:22,397 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:22,397 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:22,397 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:22,399 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,399 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c5da2da[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:22,401 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:22,401 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,402 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:22,402 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:22,403 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,403 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:22,405 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,405 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:22,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:22,409 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:22,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:22,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:22,409 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:22,409 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:22,411 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,412 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,413 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:22,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:22,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,416 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:22,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,417 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:22,417 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:22,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419442458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419442459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419442460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419442460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419442460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,517 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:22,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:22,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:22,520 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:22,520 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:22,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:22,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,521 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,522 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,523 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,523 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:22,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:22,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:22,527 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,527 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:22,527 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:22,527 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:22,529 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8020c1[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:22,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:22,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,532 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:22,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,532 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:22,533 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,533 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,534 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:22,536 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:22,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:22,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:22,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:22,536 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:22,537 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,537 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:22,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,537 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,538 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:22,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,539 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,547 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:22,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:22,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:22,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,547 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:22,547 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:22,565 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041333cf87f925dd4dcd8476ad3393274437 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041333cf87f925dd4dcd8476ad3393274437 2023-04-13 20:56:22,570 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1598, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/283133053d8c499596178526b22d77c4 2023-04-13 20:56:22,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1598 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/9d83033ebea548bfa10d811d7d172ec5 2023-04-13 20:56:22,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1598 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/1887f990f0e24928ae370f3d1e05d492 2023-04-13 20:56:22,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/283133053d8c499596178526b22d77c4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4 2023-04-13 20:56:22,596 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4, entries=200, sequenceid=1598, filesize=32.3 K 2023-04-13 20:56:22,597 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/9d83033ebea548bfa10d811d7d172ec5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5 2023-04-13 20:56:22,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5, entries=150, sequenceid=1598, filesize=11.9 K 2023-04-13 20:56:22,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/1887f990f0e24928ae370f3d1e05d492 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492 2023-04-13 20:56:22,603 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492, entries=150, sequenceid=1598, filesize=11.9 K 2023-04-13 20:56:22,604 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~325.17 KB/332976, currentSize=73.80 KB/75570 for b19b9c1a7b3fa24b26a81e15015b427d in 468ms, sequenceid=1598, compaction requested=true 2023-04-13 20:56:22,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:22,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:22,604 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:22,604 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:22,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:22,604 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:22,605 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:22,605 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46472 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:22,606 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,606 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,606 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/6091ee476c9d45d79f7dded201a3ce12, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=108.8 K 2023-04-13 20:56:22,606 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/077b6e6ff527442186b98ab06e8d9281, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=45.4 K 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 6091ee476c9d45d79f7dded201a3ce12, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1681419379581 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 077b6e6ff527442186b98ab06e8d9281, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=1532, earliestPutTs=1681419379581 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 331723f764cb464e856f4b60839b4f55, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1550, earliestPutTs=1681419379899 2023-04-13 20:56:22,606 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 612461aab0ed48d385a41a3a1ce693bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1550, earliestPutTs=1681419379899 2023-04-13 20:56:22,607 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0a551e37095143b79f01b230b4dab5d7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1564, earliestPutTs=1681419382048 2023-04-13 20:56:22,607 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7029a2a7790c4edcb8241f9194082153, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1564, earliestPutTs=1681419382049 2023-04-13 20:56:22,607 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 283133053d8c499596178526b22d77c4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:22,607 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 9d83033ebea548bfa10d811d7d172ec5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:22,614 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#608 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:22,621 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#609 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:22,648 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:22,648 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/ed52215047394c4eb34168a16fc57730 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ed52215047394c4eb34168a16fc57730 2023-04-13 20:56:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:22,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:22,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:22,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:22,654 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:22,654 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,654 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:22,654 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:22,654 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into ed52215047394c4eb34168a16fc57730(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:22,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:22,655 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=12, startTime=1681419382604; duration=0sec 2023-04-13 20:56:22,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:22,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:22,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,655 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34320 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:22,656 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:22,656 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,656 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/978b20e090104056990883adb90ab67d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=33.5 K 2023-04-13 20:56:22,656 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 978b20e090104056990883adb90ab67d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=1550, earliestPutTs=1681419379899 2023-04-13 20:56:22,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:22,657 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8ca75e670e444cfe924b9302f4618077, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1564, earliestPutTs=1681419382049 2023-04-13 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,657 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,657 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 1887f990f0e24928ae370f3d1e05d492, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:22,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:22,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:22,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:22,659 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:22,659 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,659 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:22,660 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:22,665 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:22,703 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304131cc53684568a43ea8c24c267a8e04316 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131cc53684568a43ea8c24c267a8e04316 2023-04-13 20:56:22,709 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1613, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/958cbcde9cfe49698476cdb15577af8d 2023-04-13 20:56:22,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:22,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419442773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419442773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419442774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419442774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419442774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419442877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419442877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419442878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419442878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:22,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419442878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,071 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/640e634cbf1448d3b6dbeb557850849b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/640e634cbf1448d3b6dbeb557850849b 2023-04-13 20:56:23,075 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into 640e634cbf1448d3b6dbeb557850849b(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:23,075 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:23,075 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=12, startTime=1681419382604; duration=0sec 2023-04-13 20:56:23,075 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:23,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419443079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419443079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419443080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419443081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,083 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419443082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,094 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/5376e7e778a54edbbe9d55b8dde09559 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/5376e7e778a54edbbe9d55b8dde09559 2023-04-13 20:56:23,098 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into 5376e7e778a54edbbe9d55b8dde09559(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:23,098 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:23,098 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=13, startTime=1681419382604; duration=0sec 2023-04-13 20:56:23,098 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:23,122 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1613 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/413647e8933b423b807042187f2d411b 2023-04-13 20:56:23,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419443382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419443382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419443382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419443385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419443386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,531 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1613 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/cd4377ad2a274ed1af9f87eca590bd89 2023-04-13 20:56:23,535 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/958cbcde9cfe49698476cdb15577af8d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d 2023-04-13 20:56:23,538 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d, entries=150, sequenceid=1613, filesize=25.4 K 2023-04-13 20:56:23,539 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/413647e8933b423b807042187f2d411b as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b 2023-04-13 20:56:23,542 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b, entries=150, sequenceid=1613, filesize=11.9 K 2023-04-13 20:56:23,543 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/cd4377ad2a274ed1af9f87eca590bd89 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89 2023-04-13 20:56:23,546 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89, entries=150, sequenceid=1613, filesize=11.9 K 2023-04-13 20:56:23,547 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b19b9c1a7b3fa24b26a81e15015b427d in 888ms, sequenceid=1613, compaction requested=false 2023-04-13 20:56:23,547 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:23,547 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:23,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:23,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c18fea5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:23,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:23,551 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,552 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:23,552 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:23,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:23,557 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:23,557 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:23,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:23,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:23,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:23,560 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:23,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:23,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:23,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:23,561 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:23,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,562 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:23,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:23,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,565 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:23,568 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:23,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:23,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:23,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:23,568 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:23,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:23,568 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:23,568 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,569 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,569 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:23,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:23,569 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,669 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:23,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:23,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:23,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:23,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:23,672 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:23,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:23,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:23,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:23,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,672 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:23,675 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:23,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:23,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:23,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,675 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:23,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:23,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:23,676 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:23,677 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:23,677 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:23,677 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:23,695 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304130034ff8e655f446985f261a8f1e9aa06 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130034ff8e655f446985f261a8f1e9aa06 2023-04-13 20:56:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:23,887 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419443896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 361 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419443896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419443896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419443901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:23,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419443901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419444002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 363 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419444002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419444002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419444006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419444007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,104 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1633, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/a51d31566bb14107be67fc17c65c16c7 2023-04-13 20:56:24,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 365 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419444205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,207 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419444205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419444205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419444210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419444210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419444508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,509 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419444509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419444510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419444513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,516 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1633 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/eb7bf54fcd0a44a59c29b669999bcad9 2023-04-13 20:56:24,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:24,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419444515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,525 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1633 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/4e9b213a3d454762af44ac1abc85dfa2 2023-04-13 20:56:24,529 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/a51d31566bb14107be67fc17c65c16c7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7 2023-04-13 20:56:24,533 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7, entries=150, sequenceid=1633, filesize=25.4 K 2023-04-13 20:56:24,534 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/eb7bf54fcd0a44a59c29b669999bcad9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9 2023-04-13 20:56:24,537 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9, entries=150, sequenceid=1633, filesize=11.9 K 2023-04-13 20:56:24,538 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/4e9b213a3d454762af44ac1abc85dfa2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2 2023-04-13 20:56:24,541 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2, entries=150, sequenceid=1633, filesize=11.9 K 2023-04-13 20:56:24,542 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for b19b9c1a7b3fa24b26a81e15015b427d in 865ms, sequenceid=1633, compaction requested=true 2023-04-13 20:56:24,542 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:24,542 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:24,542 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,544 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:24,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:24,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ebb051[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:24,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:24,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,547 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:24,547 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:24,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,548 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:24,549 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:24,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:24,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:24,552 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:24,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:24,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:24,552 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:24,552 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:24,555 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,556 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:24,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,560 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,560 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,560 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:24,560 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:24,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,660 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:24,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:24,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:24,664 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,664 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,666 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:24,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,667 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:24,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:24,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,667 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:24,668 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:24,668 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,668 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:24,668 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:56:24,668 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:24,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:24,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:24,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:24,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:24,669 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:24,685 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041306049ba412be45918449ac6f3f822c8c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306049ba412be45918449ac6f3f822c8c 2023-04-13 20:56:24,691 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1648, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/aa7208b9db00407caf9c142724d99192 2023-04-13 20:56:24,701 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1648 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/122a52bcaad342909d833c81dcf5e6e5 2023-04-13 20:56:24,710 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1648 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/53d5a4f064c9444ebbc212e47dccb8ab 2023-04-13 20:56:24,714 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/aa7208b9db00407caf9c142724d99192 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192 2023-04-13 20:56:24,717 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192, entries=150, sequenceid=1648, filesize=25.4 K 2023-04-13 20:56:24,718 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/122a52bcaad342909d833c81dcf5e6e5 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5 2023-04-13 20:56:24,721 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5, entries=150, sequenceid=1648, filesize=11.9 K 2023-04-13 20:56:24,722 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/53d5a4f064c9444ebbc212e47dccb8ab as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab 2023-04-13 20:56:24,729 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab, entries=150, sequenceid=1648, filesize=11.9 K 2023-04-13 20:56:24,730 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for b19b9c1a7b3fa24b26a81e15015b427d in 62ms, sequenceid=1648, compaction requested=true 2023-04-13 20:56:24,730 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:24,730 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:24,730 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:24,732 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,732 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:24,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16918d04[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:24,734 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:24,734 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,736 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:24,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,736 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,738 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,738 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:24,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:24,740 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:24,740 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:24,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:24,741 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:24,741 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:24,742 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,742 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,742 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,743 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,744 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,747 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:24,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,747 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,747 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:24,748 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,848 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:24,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:24,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:24,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:24,851 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:24,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:24,851 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:24,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,854 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,854 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,854 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:24,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:24,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:24,856 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,856 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:24,856 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:24,856 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:24,856 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:24,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,858 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,859 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:24,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5859213f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:24,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:24,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,862 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:24,862 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:24,862 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,862 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:24,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:24,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,864 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:24,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:24,867 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:24,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:24,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:24,867 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:24,867 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:24,869 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,869 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,869 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,870 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,871 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,877 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:24,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:24,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,878 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:24,878 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:24,877 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,878 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,878 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,878 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,978 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:24,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:24,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:24,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:24,981 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:24,981 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,981 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:24,982 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:24,982 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,982 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,983 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:24,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:24,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:24,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,984 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:24,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:24,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:24,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:24,985 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:24,985 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,985 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:24,986 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:24,986 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:24,986 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:24,987 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,988 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:24,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef1147[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:24,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:24,989 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,992 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:24,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:24,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:24,992 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:24,992 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,992 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:24,993 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,993 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:24,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:24,996 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:24,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:24,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:24,996 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:24,996 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:24,998 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,998 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:24,998 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:24,998 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,000 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,002 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:25,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:25,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:25,002 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,003 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,003 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:25,003 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:25,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:25,036 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:25,037 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:25,050 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304138cc29a9265fc49d1b4a59ceec6ea471b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138cc29a9265fc49d1b4a59ceec6ea471b 2023-04-13 20:56:25,054 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1661, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/995c63b4db2d4ef29f78866f04ebead6 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419385055, max timestamp=1681419385060Num uniques 859; 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419385055, max timestamp=1681419385060Num uniques 859; 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1681419385055, max timestamp=1681419385060Num uniques 859; 2023-04-13 20:56:25,063 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-13 20:56:25,066 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1661 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/5a8338b836a44243a942036056d58d32 2023-04-13 20:56:25,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419445075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419445076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419445080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419445081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419445081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,103 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:25,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:25,106 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:25,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:25,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:25,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,106 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,109 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,109 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:25,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:25,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:25,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:25,110 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:25,110 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,110 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:25,110 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:25,111 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:25,113 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,113 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79fc4053[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:25,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:25,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,116 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,116 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:25,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,117 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:25,117 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:25,118 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:25,121 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:25,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:25,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:25,121 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:25,121 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:25,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,124 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,125 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:25,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:25,128 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,128 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,128 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,128 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419445182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419445182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419445185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419445187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419445187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,228 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:25,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:25,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:25,232 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,232 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,234 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:25,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:25,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:25,236 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,236 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:25,236 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:25,236 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:25,236 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,238 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:25,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@639f1bc5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:25,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:25,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,242 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:25,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:25,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:25,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,244 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:25,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:25,247 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:25,247 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:25,247 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:25,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,249 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,250 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,251 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:25,256 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,256 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:25,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:25,256 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,257 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:25,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,257 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,357 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:25,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:25,360 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:25,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:25,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:25,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,360 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,362 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,362 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,363 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,363 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:25,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:25,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:25,364 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:25,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,366 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:25,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725e8e50[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:25,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:25,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,369 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:25,370 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:25,370 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,370 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:25,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,372 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:25,375 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:25,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:25,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:25,375 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:25,375 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:25,376 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,376 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,377 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,377 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,378 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,380 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,381 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,381 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:25,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:25,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,381 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,381 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:25,381 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:25,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419445385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419445386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419445389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419445390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,394 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419445391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1661 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/127b98ce00854cd685777703ec93e5bd 2023-04-13 20:56:25,479 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/995c63b4db2d4ef29f78866f04ebead6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6 2023-04-13 20:56:25,481 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:25,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6, entries=400, sequenceid=1661, filesize=59.8 K 2023-04-13 20:56:25,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/5a8338b836a44243a942036056d58d32 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32 2023-04-13 20:56:25,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:25,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:25,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:25,484 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:25,484 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:25,485 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,485 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,486 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,486 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32, entries=150, sequenceid=1661, filesize=11.9 K 2023-04-13 20:56:25,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:25,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/127b98ce00854cd685777703ec93e5bd as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd 2023-04-13 20:56:25,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:25,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:25,488 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:25,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:25,490 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd, entries=150, sequenceid=1661, filesize=11.9 K 2023-04-13 20:56:25,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for b19b9c1a7b3fa24b26a81e15015b427d in 454ms, sequenceid=1661, compaction requested=true 2023-04-13 20:56:25,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:25,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:25,490 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:25,490 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:25,491 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:25,491 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-13 20:56:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,491 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,491 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:25,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@940cc75[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:25,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:25,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,493 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 165746 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:25,493 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:25,493 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,493 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/640e634cbf1448d3b6dbeb557850849b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=161.9 K 2023-04-13 20:56:25,494 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61238 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:25,494 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:25,494 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,494 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ed52215047394c4eb34168a16fc57730, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=59.8 K 2023-04-13 20:56:25,494 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 640e634cbf1448d3b6dbeb557850849b, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:25,494 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ed52215047394c4eb34168a16fc57730, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:25,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:25,494 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:25,495 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 958cbcde9cfe49698476cdb15577af8d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1613, earliestPutTs=1681419382144 2023-04-13 20:56:25,495 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:25,495 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 413647e8933b423b807042187f2d411b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1613, earliestPutTs=1681419382144 2023-04-13 20:56:25,495 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,495 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:25,495 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting eb7bf54fcd0a44a59c29b669999bcad9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1633, earliestPutTs=1681419382772 2023-04-13 20:56:25,495 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting a51d31566bb14107be67fc17c65c16c7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1633, earliestPutTs=1681419382772 2023-04-13 20:56:25,496 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 122a52bcaad342909d833c81dcf5e6e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1648, earliestPutTs=1681419383894 2023-04-13 20:56:25,496 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting aa7208b9db00407caf9c142724d99192, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1648, earliestPutTs=1681419383894 2023-04-13 20:56:25,496 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,496 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5a8338b836a44243a942036056d58d32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385032 2023-04-13 20:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,496 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 995c63b4db2d4ef29f78866f04ebead6, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385027 2023-04-13 20:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:25,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:25,499 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:25,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:25,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:25,499 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:25,499 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:25,501 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,501 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,501 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:25,503 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:25,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,505 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:25,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:25,507 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#623 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,508 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:25,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:25,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:25,508 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,508 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:25,508 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:25,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:25,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:25,515 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#624 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:25,609 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:25,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:25,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:25,612 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:25,612 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,612 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:25,612 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:25,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,613 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,614 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:25,614 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,615 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:25,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:25,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:25,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:25,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:25,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:25,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:25,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:25,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:25,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:25,616 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-04-13 20:56:25,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:25,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:25,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:25,617 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:25,617 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:25,617 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:25,635 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041321ca15f34a5f4537a5e9cc269332df3e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041321ca15f34a5f4537a5e9cc269332df3e 2023-04-13 20:56:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:25,692 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:25,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419445698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419445701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419445701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419445702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419445703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419445803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419445806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419445806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419445806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:25,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419445806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:25,916 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/83af0b8771b749e990087b933def1345 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/83af0b8771b749e990087b933def1345 2023-04-13 20:56:25,920 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into 83af0b8771b749e990087b933def1345(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:25,920 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:25,920 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=11, startTime=1681419385490; duration=0sec 2023-04-13 20:56:25,920 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:25,920 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:25,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61238 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:25,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:25,922 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:25,922 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/5376e7e778a54edbbe9d55b8dde09559, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=59.8 K 2023-04-13 20:56:25,922 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 5376e7e778a54edbbe9d55b8dde09559, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=1598, earliestPutTs=1681419382090 2023-04-13 20:56:25,923 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cd4377ad2a274ed1af9f87eca590bd89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1613, earliestPutTs=1681419382144 2023-04-13 20:56:25,923 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 4e9b213a3d454762af44ac1abc85dfa2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1633, earliestPutTs=1681419382772 2023-04-13 20:56:25,924 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 53d5a4f064c9444ebbc212e47dccb8ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1648, earliestPutTs=1681419383894 2023-04-13 20:56:25,924 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 127b98ce00854cd685777703ec93e5bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385032 2023-04-13 20:56:25,934 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#626 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:25,936 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/db893706695c440a9914753fe57a2be0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/db893706695c440a9914753fe57a2be0 2023-04-13 20:56:25,940 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into db893706695c440a9914753fe57a2be0(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:25,940 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:25,940 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=11, startTime=1681419385490; duration=0sec 2023-04-13 20:56:25,940 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:25,943 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/e00d7fe8c21c46b9be272d9b47164c11 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e00d7fe8c21c46b9be272d9b47164c11 2023-04-13 20:56:25,946 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into e00d7fe8c21c46b9be272d9b47164c11(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:25,946 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:25,946 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=11, startTime=1681419385491; duration=0sec 2023-04-13 20:56:25,946 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:26,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419446006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419446009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419446010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419446010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419446010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,040 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1696, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/7602f9895dc24acdaf2d78be1c1d5653 2023-04-13 20:56:26,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419446309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419446312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419446314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419446314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419446314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,452 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1696 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/3e6b9bcb01ec4811b5b8d3fba651498e 2023-04-13 20:56:26,461 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1696 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/71578e2a8e974a82b79c657f25fdbed1 2023-04-13 20:56:26,464 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/7602f9895dc24acdaf2d78be1c1d5653 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653 2023-04-13 20:56:26,467 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653, entries=150, sequenceid=1696, filesize=25.4 K 2023-04-13 20:56:26,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/3e6b9bcb01ec4811b5b8d3fba651498e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e 2023-04-13 20:56:26,471 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e, entries=150, sequenceid=1696, filesize=11.9 K 2023-04-13 20:56:26,472 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/71578e2a8e974a82b79c657f25fdbed1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1 2023-04-13 20:56:26,474 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1, entries=150, sequenceid=1696, filesize=11.9 K 2023-04-13 20:56:26,475 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=53.67 KB/54960 for b19b9c1a7b3fa24b26a81e15015b427d in 859ms, sequenceid=1696, compaction requested=false 2023-04-13 20:56:26,475 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:26,475 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:26,475 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,477 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,477 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:26,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1225ebd6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:26,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:26,479 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,480 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:26,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:26,480 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:26,483 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:26,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:26,486 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:26,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:26,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:26,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:26,486 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:26,488 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,488 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,488 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,489 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,491 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,493 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:26,493 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:26,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:26,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:26,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:26,494 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:26,494 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:26,494 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:26,494 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,594 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:26,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:26,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:26,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:26,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:26,598 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:26,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:26,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:26,598 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:26,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:26,603 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:26,603 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:26,604 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:26,604 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:26,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:26,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:26,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:26,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:26,605 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:26,605 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:26,606 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:26,606 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:26,621 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304131febbd98b23a44c2bc32b3e96cf6c8cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131febbd98b23a44c2bc32b3e96cf6c8cb 2023-04-13 20:56:26,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:26,817 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419446834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419446835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419446836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419386831, max timestamp=1681419386836Num uniques 859; 2023-04-13 20:56:26,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:26,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419386831, max timestamp=1681419386836Num uniques 859; 2023-04-13 20:56:26,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:26,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419386831, max timestamp=1681419386836Num uniques 859; 2023-04-13 20:56:26,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:26,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419446871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419446875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419446939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419446939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 399 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419446940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419446976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:26,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:26,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419446979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,027 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1710, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/bab0397aa7854864967bb6334a4c5b54 2023-04-13 20:56:27,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419447142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419447143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 401 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419447143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,181 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419447178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419447182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,437 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1710 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/cf6ea963d2cd4e3899afc426320e2910 2023-04-13 20:56:27,447 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1710 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/6f5c4ef7a7654235be443a2498cf4f0a 2023-04-13 20:56:27,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419447446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419447447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 403 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419447448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,452 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/bab0397aa7854864967bb6334a4c5b54 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54 2023-04-13 20:56:27,456 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54, entries=150, sequenceid=1710, filesize=25.4 K 2023-04-13 20:56:27,456 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/cf6ea963d2cd4e3899afc426320e2910 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910 2023-04-13 20:56:27,459 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910, entries=150, sequenceid=1710, filesize=11.9 K 2023-04-13 20:56:27,460 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/6f5c4ef7a7654235be443a2498cf4f0a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a 2023-04-13 20:56:27,463 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a, entries=150, sequenceid=1710, filesize=11.9 K 2023-04-13 20:56:27,464 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for b19b9c1a7b3fa24b26a81e15015b427d in 858ms, sequenceid=1710, compaction requested=true 2023-04-13 20:56:27,464 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:27,464 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:27,464 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:27,466 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:27,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97f8b64[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:27,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:27,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:27,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:27,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:27,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:27,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:27,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:27,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:27,473 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:27,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:27,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:27,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:27,474 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:27,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:27,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:27,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,482 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:27,482 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:27,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:27,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419386865, max timestamp=1681419387485Num uniques 1018; 2023-04-13 20:56:27,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:56:27,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:27,487 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-13 20:56:27,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419447514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419447514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,582 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:27,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:27,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:27,585 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:27,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:27,585 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:27,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,587 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,587 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:27,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:27,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:27,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:27,589 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:27,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:27,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,591 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,591 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:27,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd5b815[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:27,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:27,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:27,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:27,596 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:27,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,596 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:27,597 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:27,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,597 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:27,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:27,599 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:27,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:27,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:27,599 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:27,599 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:27,601 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,602 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:27,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:27,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,607 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,608 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:27,608 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,608 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:27,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419447618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,619 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419447618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,708 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:27,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:27,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:27,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:27,711 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:27,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:27,711 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:27,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,712 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,716 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:27,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:27,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:27,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:27,718 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,718 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:27,718 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:27,718 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:27,718 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:27,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f9fefe[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:27,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:27,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,723 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:27,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:27,724 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:27,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,724 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:27,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:27,727 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:27,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:27,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:27,727 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:27,727 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:27,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,729 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,730 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,731 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:27,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:27,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,734 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:27,734 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,734 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,734 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,734 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,734 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:27,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419447821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419447821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,834 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:27,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:27,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:27,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:27,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:27,837 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:27,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:27,837 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:27,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,838 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,839 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,839 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:27,840 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:27,840 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:27,841 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:27,841 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:27,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,849 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,849 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:27,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d3fde6c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:27,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:27,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,851 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:27,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:27,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:27,853 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,853 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:27,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:27,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:27,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:27,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:27,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:27,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:27,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,859 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,860 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,861 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,863 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:27,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:27,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,864 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,864 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:27,864 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:27,905 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413d82150c7d3734e478474a4bde3e3de13 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d82150c7d3734e478474a4bde3e3de13 2023-04-13 20:56:27,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419447950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419447953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:27,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 405 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419447953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,964 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:27,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:27,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:27,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:27,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:27,968 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:27,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:27,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:27,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,970 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:27,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:27,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:27,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:27,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:27,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:27,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:27,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:27,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:27,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:27,974 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,974 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,974 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,974 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:27,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:27,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486e4604[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:27,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:27,977 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:27,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:27,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:27,978 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,978 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:27,980 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,980 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:27,981 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,981 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:27,983 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:27,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:27,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:27,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:27,983 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:27,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:27,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:27,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:27,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,986 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:27,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:27,990 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:27,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:27,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:27,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:27,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:27,990 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:27,991 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:28,091 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:28,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:28,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:28,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:28,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:28,094 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:28,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:28,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:28,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:28,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:28,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:28,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:28,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:28,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:28,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:28,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:28,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:28,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:28,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:28,101 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,101 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:28,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:28,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad95a0c[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:28,103 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:28,103 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,104 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,104 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:28,104 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:28,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:28,105 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,105 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:28,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:28,106 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,106 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:28,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:28,108 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:28,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:28,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:28,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:28,108 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:28,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:28,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:28,111 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,112 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:28,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,115 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:28,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:28,115 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:28,115 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:28,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:28,115 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:28,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:28,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:28,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419448126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419448127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,216 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:28,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:28,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:28,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:28,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:28,219 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:28,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:28,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:28,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:28,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:28,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,221 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:28,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:28,222 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:28,222 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:28,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:28,223 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:28,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:28,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:28,223 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:28,224 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,224 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,225 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:28,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251857a7[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:28,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:28,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:28,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:28,228 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:28,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:28,233 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,233 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:28,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:28,236 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:28,236 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:28,236 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:28,236 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:28,236 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:28,238 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,238 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:28,238 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:28,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:28,239 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:28,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:28,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:28,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:28,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:28,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:28,243 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:28,244 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:28,244 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:28,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:28,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:28,244 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:28,310 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1748, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/3372285868b9484791ada4b1ca116822 2023-04-13 20:56:28,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1748 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/762052d8893142ed83034855a1ec2dc8 2023-04-13 20:56:28,328 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1748 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/db81659f487f497196b3a8647f13cd29 2023-04-13 20:56:28,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/3372285868b9484791ada4b1ca116822 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822 2023-04-13 20:56:28,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822, entries=200, sequenceid=1748, filesize=32.3 K 2023-04-13 20:56:28,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/762052d8893142ed83034855a1ec2dc8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8 2023-04-13 20:56:28,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8, entries=150, sequenceid=1748, filesize=11.9 K 2023-04-13 20:56:28,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/db81659f487f497196b3a8647f13cd29 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29 2023-04-13 20:56:28,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29, entries=150, sequenceid=1748, filesize=11.9 K 2023-04-13 20:56:28,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~292.16 KB/299168, currentSize=80.51 KB/82440 for b19b9c1a7b3fa24b26a81e15015b427d in 855ms, sequenceid=1748, compaction requested=true 2023-04-13 20:56:28,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:28,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:28,342 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:28,342 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:28,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:28,342 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:28,343 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111702 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:28,343 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:28,343 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:28,343 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:28,344 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,344 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,344 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/db893706695c440a9914753fe57a2be0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=109.1 K 2023-04-13 20:56:28,344 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/83af0b8771b749e990087b933def1345, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=48.1 K 2023-04-13 20:56:28,344 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:28,344 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting db893706695c440a9914753fe57a2be0, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385032 2023-04-13 20:56:28,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:28,345 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 83af0b8771b749e990087b933def1345, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385032 2023-04-13 20:56:28,345 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7602f9895dc24acdaf2d78be1c1d5653, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1696, earliestPutTs=1681419385079 2023-04-13 20:56:28,345 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3e6b9bcb01ec4811b5b8d3fba651498e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1696, earliestPutTs=1681419385079 2023-04-13 20:56:28,345 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting cf6ea963d2cd4e3899afc426320e2910, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1710, earliestPutTs=1681419385699 2023-04-13 20:56:28,345 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting bab0397aa7854864967bb6334a4c5b54, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1710, earliestPutTs=1681419385699 2023-04-13 20:56:28,346 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 3372285868b9484791ada4b1ca116822, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386865 2023-04-13 20:56:28,346 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 762052d8893142ed83034855a1ec2dc8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386869 2023-04-13 20:56:28,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:28,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:28,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:28,348 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:28,348 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,348 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:28,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:28,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,351 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:28,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:28,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:28,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:28,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:28,352 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:28,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:28,366 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:28,367 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:28,367 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:28,367 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:28,369 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#636 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:28,372 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#635 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:28,411 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/0c0b148d03b84db58c11b8cea3c3593e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0c0b148d03b84db58c11b8cea3c3593e 2023-04-13 20:56:28,415 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into 0c0b148d03b84db58c11b8cea3c3593e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:28,415 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:28,415 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=12, startTime=1681419388342; duration=0sec 2023-04-13 20:56:28,415 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:28,415 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:28,416 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:28,417 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:28,417 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:28,417 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e00d7fe8c21c46b9be272d9b47164c11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=48.1 K 2023-04-13 20:56:28,417 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting e00d7fe8c21c46b9be272d9b47164c11, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1661, earliestPutTs=1681419385032 2023-04-13 20:56:28,418 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 71578e2a8e974a82b79c657f25fdbed1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1696, earliestPutTs=1681419385079 2023-04-13 20:56:28,418 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 6f5c4ef7a7654235be443a2498cf4f0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1710, earliestPutTs=1681419385699 2023-04-13 20:56:28,418 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting db81659f487f497196b3a8647f13cd29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386869 2023-04-13 20:56:28,425 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#638 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:28,440 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/0c597546fe2e4f16b0199fa62170e6c4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0c597546fe2e4f16b0199fa62170e6c4 2023-04-13 20:56:28,445 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into 0c597546fe2e4f16b0199fa62170e6c4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:28,445 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:28,445 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=12, startTime=1681419388342; duration=0sec 2023-04-13 20:56:28,445 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:28,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:28,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419448651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419448653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419448755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419448755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,805 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413f10b79ae413d4ddf96b46e4454c134cf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f10b79ae413d4ddf96b46e4454c134cf 2023-04-13 20:56:28,826 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/0de5ce0864a94f7bb3ecf87f31ace0bf as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0de5ce0864a94f7bb3ecf87f31ace0bf 2023-04-13 20:56:28,830 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into 0de5ce0864a94f7bb3ecf87f31ace0bf(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:28,830 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:28,830 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=12, startTime=1681419388342; duration=0sec 2023-04-13 20:56:28,830 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:28,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419448952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419448955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,960 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419448958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419448958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:28,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419448965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,218 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1763, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/720c627ac111417bbeaceb40bb506790 2023-04-13 20:56:29,228 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1763 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/fef45f65ca7a48c3b798a3fb87e1c8a2 2023-04-13 20:56:29,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419449262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,264 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419449262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,638 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1763 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/f64a053975ec4caca22eaabe481222df 2023-04-13 20:56:29,642 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/720c627ac111417bbeaceb40bb506790 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790 2023-04-13 20:56:29,646 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790, entries=150, sequenceid=1763, filesize=25.4 K 2023-04-13 20:56:29,646 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/fef45f65ca7a48c3b798a3fb87e1c8a2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2 2023-04-13 20:56:29,650 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2, entries=150, sequenceid=1763, filesize=11.9 K 2023-04-13 20:56:29,650 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/f64a053975ec4caca22eaabe481222df as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df 2023-04-13 20:56:29,653 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df, entries=150, sequenceid=1763, filesize=11.9 K 2023-04-13 20:56:29,654 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b19b9c1a7b3fa24b26a81e15015b427d in 1287ms, sequenceid=1763, compaction requested=false 2023-04-13 20:56:29,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:29,654 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:29,657 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:29,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@669731d6[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:29,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:29,659 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,660 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:29,660 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:29,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,661 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,662 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,662 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:29,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:29,666 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:29,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:29,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:29,666 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:29,666 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:29,667 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,668 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,670 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,682 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,683 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,683 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:29,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:29,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:29,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:29,683 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:29,683 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:29,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:29,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:29,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:29,778 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413fbc524bd0685434b9a4f892c0c499421 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fbc524bd0685434b9a4f892c0c499421 2023-04-13 20:56:29,783 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1784, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/b04599fca5194f3aa70ee6b8c6a048f2 2023-04-13 20:56:29,784 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:29,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:29,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:29,787 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:29,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:29,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:29,787 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:29,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,788 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,789 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:29,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:29,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,790 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:29,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:29,791 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:29,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:29,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:29,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:29,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:29,791 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:29,791 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:29,791 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:29,792 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:29,792 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:29,793 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,793 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419449791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,796 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419449793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c731566[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:29,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:29,797 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,798 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:29,799 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:29,799 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,799 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:29,801 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:29,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,801 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:29,804 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:29,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:29,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:29,804 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:29,804 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:29,806 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,806 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,807 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,810 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:29,810 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:29,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:29,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:29,810 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,810 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,811 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,811 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,811 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419449895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419449897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,911 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:29,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:29,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:29,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:29,914 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:29,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:29,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:29,914 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:29,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,915 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,916 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:29,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:29,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,916 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,917 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:29,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:29,918 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:29,918 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:29,918 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:29,918 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:29,918 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:29,921 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,921 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:29,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda2dde[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:29,924 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,925 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:29,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,925 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:29,925 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:29,927 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,927 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:29,929 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:29,929 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:29,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,934 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:29,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,935 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,936 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:29,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:29,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:29,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:29,938 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:29,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:29,938 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:29,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:29,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:29,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:29,939 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:29,939 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:29,939 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:30,039 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:30,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:30,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:30,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:30,043 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:30,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:30,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:30,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,043 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,044 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:30,044 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:30,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,045 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:30,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:30,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:30,046 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:30,046 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:30,047 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:30,047 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:30,049 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,049 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b18f88d[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:30,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:30,051 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,054 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:30,054 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:30,055 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:30,054 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,055 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:30,056 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,056 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:30,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:30,059 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:30,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:30,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:30,059 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:30,059 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:30,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,060 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:30,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:30,061 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,062 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:30,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:30,064 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:30,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,065 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,065 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:30,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:30,065 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:30,065 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:30,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419450098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 453 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419450100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,165 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:30,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:30,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:30,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:30,169 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,169 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:30,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:30,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,174 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:30,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:30,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:30,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:30,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:30,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:30,175 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:30,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:30,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:30,176 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:30,176 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:30,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,178 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:30,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@445e73b9[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:30,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:30,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,182 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,182 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:30,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:30,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:30,182 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:30,183 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:30,183 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:30,183 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,184 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:30,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:30,186 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:30,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:30,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:30,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:30,187 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:30,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:30,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:30,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,189 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,189 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:30,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:30,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,190 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1784 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/bb0da5eec92f48a896a95989daebf184 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:30,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:30,194 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:30,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:30,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:30,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,194 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:30,194 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:30,194 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:30,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1784 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/674a880b766349fdaf64d15f1ff0ad7a 2023-04-13 20:56:30,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/b04599fca5194f3aa70ee6b8c6a048f2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2 2023-04-13 20:56:30,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2, entries=200, sequenceid=1784, filesize=32.3 K 2023-04-13 20:56:30,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/bb0da5eec92f48a896a95989daebf184 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184 2023-04-13 20:56:30,216 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184, entries=150, sequenceid=1784, filesize=11.9 K 2023-04-13 20:56:30,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/674a880b766349fdaf64d15f1ff0ad7a as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a 2023-04-13 20:56:30,221 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a, entries=150, sequenceid=1784, filesize=11.9 K 2023-04-13 20:56:30,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b19b9c1a7b3fa24b26a81e15015b427d in 455ms, sequenceid=1784, compaction requested=true 2023-04-13 20:56:30,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:30,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:30,222 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:30,222 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:30,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:30,222 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:30,223 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85792 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:30,223 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:30,223 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:30,223 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:30,223 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,223 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,223 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0de5ce0864a94f7bb3ecf87f31ace0bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=83.8 K 2023-04-13 20:56:30,223 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0c0b148d03b84db58c11b8cea3c3593e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=36.3 K 2023-04-13 20:56:30,224 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0de5ce0864a94f7bb3ecf87f31ace0bf, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386869 2023-04-13 20:56:30,224 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0c0b148d03b84db58c11b8cea3c3593e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386869 2023-04-13 20:56:30,224 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 720c627ac111417bbeaceb40bb506790, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1763, earliestPutTs=1681419387497 2023-04-13 20:56:30,224 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting fef45f65ca7a48c3b798a3fb87e1c8a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1763, earliestPutTs=1681419387497 2023-04-13 20:56:30,225 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting bb0da5eec92f48a896a95989daebf184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388650 2023-04-13 20:56:30,225 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting b04599fca5194f3aa70ee6b8c6a048f2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388649 2023-04-13 20:56:30,231 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#644 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:30,235 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#645 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:30,241 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/3351a0e5715f47f9bb8e1c6b547f29fe as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3351a0e5715f47f9bb8e1c6b547f29fe 2023-04-13 20:56:30,245 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into 3351a0e5715f47f9bb8e1c6b547f29fe(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:30,245 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:30,245 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=13, startTime=1681419390222; duration=0sec 2023-04-13 20:56:30,245 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:30,245 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-13 20:56:30,246 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-13 20:56:30,246 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:30,247 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,247 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0c597546fe2e4f16b0199fa62170e6c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=36.3 K 2023-04-13 20:56:30,247 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0c597546fe2e4f16b0199fa62170e6c4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1748, earliestPutTs=1681419386869 2023-04-13 20:56:30,247 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f64a053975ec4caca22eaabe481222df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1763, earliestPutTs=1681419387497 2023-04-13 20:56:30,248 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 674a880b766349fdaf64d15f1ff0ad7a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388650 2023-04-13 20:56:30,269 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#646 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:30,286 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/e5c419fdd94f404ea174ddb8dd7058a8 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e5c419fdd94f404ea174ddb8dd7058a8 2023-04-13 20:56:30,287 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/0cdb9d77bf464339ab93907a7e790b79 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0cdb9d77bf464339ab93907a7e790b79 2023-04-13 20:56:30,292 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into 0cdb9d77bf464339ab93907a7e790b79(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:30,292 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:30,292 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=13, startTime=1681419390222; duration=0sec 2023-04-13 20:56:30,292 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into e5c419fdd94f404ea174ddb8dd7058a8(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:30,292 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:30,292 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=13, startTime=1681419390222; duration=0sec 2023-04-13 20:56:30,292 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:30,292 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:30,294 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:30,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:30,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:30,298 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:30,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:30,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:30,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,299 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:30,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:30,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:30,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,301 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:30,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:30,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:30,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:30,302 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:30,303 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:30,303 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:30,315 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304132cb83587a6e64463ae801d97821cab56 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132cb83587a6e64463ae801d97821cab56 2023-04-13 20:56:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:30,403 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:30,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419450428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 459 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419450429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 466 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419450531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 461 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419450532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,722 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1801, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/8bdf6ece176e4b3eb1c66c2ae3709d79 2023-04-13 20:56:30,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419450734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419450734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419450969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,972 DEBUG [Thread-4331] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:30,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419450970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,974 DEBUG [Thread-4335] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:30,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 412 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419450974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:30,977 DEBUG [Thread-4333] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:31,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419451038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419451038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,133 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1801 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/8df63e7ffe38467abb2c2bbd8ea85ba2 2023-04-13 20:56:31,143 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1801 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/560df7fd06614a908747a2f946f00b2d 2023-04-13 20:56:31,147 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/8bdf6ece176e4b3eb1c66c2ae3709d79 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79 2023-04-13 20:56:31,151 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79, entries=150, sequenceid=1801, filesize=25.4 K 2023-04-13 20:56:31,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/8df63e7ffe38467abb2c2bbd8ea85ba2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2 2023-04-13 20:56:31,157 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2, entries=150, sequenceid=1801, filesize=11.9 K 2023-04-13 20:56:31,158 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/560df7fd06614a908747a2f946f00b2d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d 2023-04-13 20:56:31,161 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d, entries=150, sequenceid=1801, filesize=11.9 K 2023-04-13 20:56:31,161 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for b19b9c1a7b3fa24b26a81e15015b427d in 858ms, sequenceid=1801, compaction requested=false 2023-04-13 20:56:31,161 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:31,162 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:31,162 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:31,164 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:31,166 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,166 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:31,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:31,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,167 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:31,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:31,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429671[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:31,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:31,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,171 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:31,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:31,171 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:31,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:31,174 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:31,174 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:31,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:31,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:31,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:31,176 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:31,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:31,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:31,176 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:31,177 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:31,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,178 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:31,179 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:31,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:31,179 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:31,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,180 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:31,187 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:31,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:31,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:31,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:31,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:31,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:31,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:31,187 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:31,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,188 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:31,188 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:31,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:31,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:31,188 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,288 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:31,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:31,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:31,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:31,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:31,291 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:31,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:31,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:31,291 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:31,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,292 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:31,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,294 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:31,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:31,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:31,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:31,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:31,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:31,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:31,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:31,296 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:31,296 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:31,296 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:31,314 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413303229c0c7d148838d39c2363131c69d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413303229c0c7d148838d39c2363131c69d 2023-04-13 20:56:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:31,544 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:31,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419451559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419451562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419451663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 475 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419451666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,719 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1819, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/03127ff119da4ce9a648f4dcb6ac4606 2023-04-13 20:56:31,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 481 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419451867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:31,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:31,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 477 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419451869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,128 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1819 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/f3455732d18a428aabc3ebaa1df76fd0 2023-04-13 20:56:32,137 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1819 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/37f2a9f0fa2449daa86856ce49502737 2023-04-13 20:56:32,140 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/03127ff119da4ce9a648f4dcb6ac4606 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606 2023-04-13 20:56:32,143 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606, entries=150, sequenceid=1819, filesize=25.4 K 2023-04-13 20:56:32,144 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/f3455732d18a428aabc3ebaa1df76fd0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0 2023-04-13 20:56:32,147 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0, entries=150, sequenceid=1819, filesize=11.9 K 2023-04-13 20:56:32,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/37f2a9f0fa2449daa86856ce49502737 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737 2023-04-13 20:56:32,151 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737, entries=150, sequenceid=1819, filesize=11.9 K 2023-04-13 20:56:32,151 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for b19b9c1a7b3fa24b26a81e15015b427d in 855ms, sequenceid=1819, compaction requested=true 2023-04-13 20:56:32,151 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:32,152 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:32,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:32,152 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:32,158 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ddbff7a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:32,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:32,162 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,163 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:32,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:32,163 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,165 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:32,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:32,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:32,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:32,168 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:32,168 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:32,171 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,171 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,171 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,172 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,173 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:32,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,177 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,176 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:32,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:32,177 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,177 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:32,177 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:32,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,193 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413357233018ff44d8393fddec05b60a4d4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413357233018ff44d8393fddec05b60a4d4 2023-04-13 20:56:32,198 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1834, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/93f03849e14040949771325f52410f3e 2023-04-13 20:56:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419452199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419452200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1834 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/ad73cef1e42647468497955942cb03d3 2023-04-13 20:56:32,277 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:32,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:32,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:32,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:32,280 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:32,280 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:32,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:32,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,281 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,282 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,283 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:32,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:32,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:32,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:32,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:32,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:32,284 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:32,284 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,286 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:32,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715bb6dc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:32,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:32,288 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,289 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:32,290 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:32,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,290 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,291 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:32,294 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:32,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:32,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:32,294 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:32,294 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:32,295 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,296 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,298 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:32,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:32,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,300 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,301 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:32,300 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,301 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:32,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,301 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419452304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419452304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,401 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:32,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:32,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:32,404 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:32,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:32,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:32,404 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,405 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,407 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,407 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:32,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:32,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:32,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:32,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:32,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,408 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:32,409 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,409 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:32,409 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:32,409 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:32,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,412 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,413 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:32,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46afc0f5[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:32,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:32,414 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,415 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:32,416 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:32,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,416 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,417 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,417 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:32,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:32,420 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:32,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:32,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:32,420 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:32,420 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:32,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,423 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,424 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,425 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,427 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:32,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,427 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,428 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:32,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:32,428 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419452506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 495 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419452514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,528 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:32,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:32,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:32,531 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:32,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:32,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:32,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,531 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,535 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,535 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,536 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:32,537 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:32,538 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,538 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:32,538 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:32,538 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,543 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524a274f[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:32,544 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:32,545 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,548 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:32,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,549 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:32,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:32,549 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:32,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:32,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,550 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,550 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:32,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:32,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:32,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:32,553 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:32,553 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:32,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,554 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:32,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:32,556 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:32,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,557 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:32,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:32,563 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:32,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:32,564 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,563 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:32,564 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:32,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:32,564 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:32,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:32,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1834 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/54a71353e4ab4a239eb56f36646d62d7 2023-04-13 20:56:32,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/93f03849e14040949771325f52410f3e as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e 2023-04-13 20:56:32,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e, entries=200, sequenceid=1834, filesize=32.3 K 2023-04-13 20:56:32,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/ad73cef1e42647468497955942cb03d3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3 2023-04-13 20:56:32,624 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3, entries=150, sequenceid=1834, filesize=11.9 K 2023-04-13 20:56:32,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/54a71353e4ab4a239eb56f36646d62d7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7 2023-04-13 20:56:32,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7, entries=150, sequenceid=1834, filesize=11.9 K 2023-04-13 20:56:32,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for b19b9c1a7b3fa24b26a81e15015b427d in 455ms, sequenceid=1834, compaction requested=true 2023-04-13 20:56:32,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:32,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:32,629 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:32,629 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:32,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:32,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:32,630 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111940 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:32,630 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:32,630 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:32,630 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:32,631 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,631 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,631 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e5c419fdd94f404ea174ddb8dd7058a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=109.3 K 2023-04-13 20:56:32,631 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3351a0e5715f47f9bb8e1c6b547f29fe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=48.3 K 2023-04-13 20:56:32,631 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting e5c419fdd94f404ea174ddb8dd7058a8, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388650 2023-04-13 20:56:32,631 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 3351a0e5715f47f9bb8e1c6b547f29fe, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388650 2023-04-13 20:56:32,631 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 8bdf6ece176e4b3eb1c66c2ae3709d79, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1801, earliestPutTs=1681419389785 2023-04-13 20:56:32,631 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 8df63e7ffe38467abb2c2bbd8ea85ba2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1801, earliestPutTs=1681419389785 2023-04-13 20:56:32,632 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 03127ff119da4ce9a648f4dcb6ac4606, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1819, earliestPutTs=1681419390422 2023-04-13 20:56:32,632 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting f3455732d18a428aabc3ebaa1df76fd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1819, earliestPutTs=1681419390422 2023-04-13 20:56:32,632 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 93f03849e14040949771325f52410f3e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:32,632 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting ad73cef1e42647468497955942cb03d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:32,646 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#657 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:32,653 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#656 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:32,664 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:32,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:32,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:32,668 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:32,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,668 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,671 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:32,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:32,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,671 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:32,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:32,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:32,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:32,673 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-13 20:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:32,674 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,674 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:32,674 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:32,680 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/d526228190814ec18711b4f06ba015c1 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d526228190814ec18711b4f06ba015c1 2023-04-13 20:56:32,685 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into d526228190814ec18711b4f06ba015c1(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:32,685 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:32,685 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=12, startTime=1681419392629; duration=0sec 2023-04-13 20:56:32,685 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:32,685 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-13 20:56:32,686 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-13 20:56:32,686 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:32,686 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:32,686 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0cdb9d77bf464339ab93907a7e790b79, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=48.3 K 2023-04-13 20:56:32,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 0cdb9d77bf464339ab93907a7e790b79, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1784, earliestPutTs=1681419388650 2023-04-13 20:56:32,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 560df7fd06614a908747a2f946f00b2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1801, earliestPutTs=1681419389785 2023-04-13 20:56:32,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 37f2a9f0fa2449daa86856ce49502737, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1819, earliestPutTs=1681419390422 2023-04-13 20:56:32,687 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 54a71353e4ab4a239eb56f36646d62d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:32,693 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413397d2d09d5744e81a6a72872256181f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413397d2d09d5744e81a6a72872256181f4 2023-04-13 20:56:32,697 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#659 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:32,705 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1851, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/61bc3b47e71448b3b684cb4e589195b2 2023-04-13 20:56:32,708 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/b643291046214144b5efc8ba76212171 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b643291046214144b5efc8ba76212171 2023-04-13 20:56:32,713 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into b643291046214144b5efc8ba76212171(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:32,713 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:32,713 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=12, startTime=1681419392629; duration=0sec 2023-04-13 20:56:32,713 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:32,716 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1851 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/aee400c6c64240e98ac689a60efbc6a2 2023-04-13 20:56:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:32,811 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:32,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 499 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419452830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419452831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 501 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419452934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:32,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:32,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419452934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,062 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/561e8ad893dc4aa3bb3904f36fa378c7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/561e8ad893dc4aa3bb3904f36fa378c7 2023-04-13 20:56:33,066 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into 561e8ad893dc4aa3bb3904f36fa378c7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:33,066 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:33,066 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=12, startTime=1681419392629; duration=0sec 2023-04-13 20:56:33,066 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:33,126 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1851 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/dfe7bae2e57941d88b3b1ec9614d1061 2023-04-13 20:56:33,130 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/61bc3b47e71448b3b684cb4e589195b2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2 2023-04-13 20:56:33,133 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2, entries=150, sequenceid=1851, filesize=25.4 K 2023-04-13 20:56:33,134 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/aee400c6c64240e98ac689a60efbc6a2 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2 2023-04-13 20:56:33,138 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2, entries=150, sequenceid=1851, filesize=11.9 K 2023-04-13 20:56:33,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:33,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 503 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419453136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:33,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 506 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419453137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,139 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/dfe7bae2e57941d88b3b1ec9614d1061 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061 2023-04-13 20:56:33,147 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061, entries=150, sequenceid=1851, filesize=11.9 K 2023-04-13 20:56:33,148 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for b19b9c1a7b3fa24b26a81e15015b427d in 475ms, sequenceid=1851, compaction requested=false 2023-04-13 20:56:33,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:33,148 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:33,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:33,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:33,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:33,148 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:33,150 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,150 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:33,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:33,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20982253[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:33,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:33,152 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,153 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:33,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:33,154 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:33,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,154 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:33,157 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,157 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:33,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:33,160 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:33,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:33,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:33,160 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:33,160 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:33,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,162 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,163 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,164 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:33,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,167 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:33,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,167 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:33,167 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,167 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:33,168 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,268 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:33,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:33,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:33,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:33,271 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:33,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:33,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:33,271 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:33,272 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,272 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,273 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,273 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:33,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:33,274 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,274 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:33,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:33,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:33,275 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:33,275 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:33,275 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:33,275 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:33,276 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,293 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413e7b8e84d9a6d4be19be2e2199612faaa to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e7b8e84d9a6d4be19be2e2199612faaa 2023-04-13 20:56:33,303 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1869, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/7b1623a7690841e495d5e559a1e028b6 2023-04-13 20:56:33,314 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1869 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/b31620e0cf164a7d904fae051c0c17be 2023-04-13 20:56:33,324 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1869 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/efe4c52a3e414048bc9209d5c79945a7 2023-04-13 20:56:33,327 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/7b1623a7690841e495d5e559a1e028b6 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6 2023-04-13 20:56:33,330 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6, entries=150, sequenceid=1869, filesize=25.4 K 2023-04-13 20:56:33,331 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/b31620e0cf164a7d904fae051c0c17be as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be 2023-04-13 20:56:33,335 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be, entries=150, sequenceid=1869, filesize=11.9 K 2023-04-13 20:56:33,335 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/efe4c52a3e414048bc9209d5c79945a7 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7 2023-04-13 20:56:33,339 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7, entries=150, sequenceid=1869, filesize=11.9 K 2023-04-13 20:56:33,339 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for b19b9c1a7b3fa24b26a81e15015b427d in 64ms, sequenceid=1869, compaction requested=true 2023-04-13 20:56:33,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:33,339 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:33,340 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:33,343 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,343 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,343 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,343 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:33,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd98dca[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:33,345 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:33,345 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:33,347 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,347 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:33,348 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,348 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:33,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:33,351 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:33,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:33,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:33,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:33,351 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:33,353 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,353 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,353 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,354 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,355 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,358 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:33,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:33,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,359 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:33,358 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:33,359 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:33,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,359 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:33,459 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:33,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:33,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:33,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:33,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:33,464 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:33,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:33,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:33,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:33,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,464 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,465 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:33,465 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:33,466 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,466 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:33,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:33,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:33,467 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:33,467 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:33,467 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:33,467 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:33,467 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:33,468 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:33,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:33,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:33,483 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304136e3bdc2406c94fb4b7aaac235aaab520 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e3bdc2406c94fb4b7aaac235aaab520 2023-04-13 20:56:33,489 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1876, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/497f254a0a52434c880e5a32587b1284 2023-04-13 20:56:33,498 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1876 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/0181c894aab3436ab5208b173511dbd3 2023-04-13 20:56:33,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419393475, max timestamp=1681419393500Num uniques 859; 2023-04-13 20:56:33,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1681419393475, max timestamp=1681419393500Num uniques 859; 2023-04-13 20:56:33,502 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:33,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1681419393493, max timestamp=1681419393500Num uniques 859; 2023-04-13 20:56:33,503 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-13 20:56:33,501 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-13 20:56:33,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419393510, max timestamp=1681419393537Num uniques 1068; 2023-04-13 20:56:33,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419393510, max timestamp=1681419393537Num uniques 1018; 2023-04-13 20:56:33,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1681419393510, max timestamp=1681419393537Num uniques 1068; 2023-04-13 20:56:33,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-13 20:56:33,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:33,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-04-13 20:56:33,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419393554, max timestamp=1681419393566Num uniques 1027; 2023-04-13 20:56:33,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419393554, max timestamp=1681419393566Num uniques 1027; 2023-04-13 20:56:33,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:56:33,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1681419393554, max timestamp=1681419393566Num uniques 1027; 2023-04-13 20:56:33,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:56:33,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-13 20:56:33,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681419393584, max timestamp=1681419393593Num uniques 1036; 2023-04-13 20:56:33,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 20:56:33,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681419393584, max timestamp=1681419393593Num uniques 1036; 2023-04-13 20:56:33,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1681419393584, max timestamp=1681419393593Num uniques 1036; 2023-04-13 20:56:33,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 20:56:33,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419393606, max timestamp=1681419393633Num uniques 1095; 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419393606, max timestamp=1681419393633Num uniques 1095; 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1681419393606, max timestamp=1681419393633Num uniques 1095; 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-13 20:56:33,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-13 20:56:33,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419393657, max timestamp=1681419393664Num uniques 1054; 2023-04-13 20:56:33,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 20:56:33,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419393657, max timestamp=1681419393664Num uniques 1054; 2023-04-13 20:56:33,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 20:56:33,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1681419393657, max timestamp=1681419393664Num uniques 1054; 2023-04-13 20:56:33,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-13 20:56:33,708 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419393698, max timestamp=1681419393706Num uniques 1013; 2023-04-13 20:56:33,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 20:56:33,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419393698, max timestamp=1681419393706Num uniques 1013; 2023-04-13 20:56:33,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1681419393698, max timestamp=1681419393706Num uniques 1013; 2023-04-13 20:56:33,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 20:56:33,710 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-13 20:56:33,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419393734, max timestamp=1681419393748Num uniques 1022; 2023-04-13 20:56:33,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419393734, max timestamp=1681419393748Num uniques 1022; 2023-04-13 20:56:33,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1681419393734, max timestamp=1681419393748Num uniques 1022; 2023-04-13 20:56:33,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 20:56:33,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 20:56:33,751 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419393769, max timestamp=1681419393781Num uniques 1031; 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419393769, max timestamp=1681419393781Num uniques 1031; 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1681419393769, max timestamp=1681419393781Num uniques 1031; 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 20:56:33,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1681419393778, max timestamp=1681419393835Num uniques 1040; 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8691, totalHeapSize=20000, min timestamp=1681419393778, max timestamp=1681419393835Num uniques 1040; 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47621, new segment data size=8691, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=8690, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8690, totalHeapSize=20000, min timestamp=1681419393778, max timestamp=1681419393835Num uniques 1040; 2023-04-13 20:56:33,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=8690, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-13 20:56:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419393859, max timestamp=1681419393873Num uniques 1049; 2023-04-13 20:56:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:56:33,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419393859, max timestamp=1681419393873Num uniques 1049; 2023-04-13 20:56:33,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:56:33,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1681419393859, max timestamp=1681419393873Num uniques 1049; 2023-04-13 20:56:33,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-04-13 20:56:33,913 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1876 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/afd8b0e8b250401a99ce223ce7e3b682 2023-04-13 20:56:33,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:33,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 613 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419453911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419393904, max timestamp=1681419393913Num uniques 1108; 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419393904, max timestamp=1681419393913Num uniques 1108; 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1681419393904, max timestamp=1681419393913Num uniques 1108; 2023-04-13 20:56:33,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-04-13 20:56:33,918 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/497f254a0a52434c880e5a32587b1284 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284 2023-04-13 20:56:33,922 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284, entries=200, sequenceid=1876, filesize=32.3 K 2023-04-13 20:56:33,923 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/0181c894aab3436ab5208b173511dbd3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3 2023-04-13 20:56:33,938 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3, entries=100, sequenceid=1876, filesize=9.5 K 2023-04-13 20:56:33,938 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/afd8b0e8b250401a99ce223ce7e3b682 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682 2023-04-13 20:56:33,943 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682, entries=100, sequenceid=1876, filesize=9.5 K 2023-04-13 20:56:33,943 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=67.09 KB/68700 for b19b9c1a7b3fa24b26a81e15015b427d in 476ms, sequenceid=1876, compaction requested=true 2023-04-13 20:56:33,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:33,943 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:33,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:33,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=73.80 KB heapSize=205.78 KB 2023-04-13 20:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:33,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:33,948 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:33,948 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,948 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:33,949 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:33,949 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-13 20:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34494ccc[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:33,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:33,954 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,956 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:33,956 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:33,957 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,957 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:33,958 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,959 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:33,962 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:33,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:33,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:33,962 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:33,962 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:33,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,964 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:33,965 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:33,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:33,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:33,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:33,968 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:33,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:33,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:33,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:33,968 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,969 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:33,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:33,969 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:33,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:33,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 633 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419454018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 616 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419454021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,069 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,073 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,073 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,075 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,076 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,077 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,077 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,077 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,078 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,078 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,078 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,079 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,079 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,080 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@373a6555[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,081 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,083 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,083 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,083 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,083 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,083 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,085 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,085 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,087 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,087 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,088 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,088 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,093 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,099 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,099 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,099 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,099 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 635 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419454122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 618 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419454126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,200 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,203 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,203 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,204 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,205 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,205 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,206 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,207 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,208 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,208 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,210 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,210 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d1535b[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,212 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,212 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,215 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,215 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,215 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,215 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,216 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,216 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,219 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,219 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,220 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,221 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,222 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,224 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,228 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,228 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,228 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,329 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 637 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419454327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,332 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,332 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] ipc.CallRunner(144): callId: 620 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419454331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,335 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,335 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,336 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,336 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,337 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,337 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,339 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,339 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a63396[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,341 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,342 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,342 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,342 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,343 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,346 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,347 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,349 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,349 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,349 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,350 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,350 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,350 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,351 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,354 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,354 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,355 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,355 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,355 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,363 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023041356a01fb122ae4d79a9e1161fdd9f1a55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a01fb122ae4d79a9e1161fdd9f1a55 2023-04-13 20:56:34,455 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,459 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,459 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,460 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,461 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,462 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,463 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,463 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,465 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,465 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,466 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68f722a2[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,468 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,469 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,469 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,469 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,470 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,471 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,471 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,474 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,474 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,474 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,476 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,481 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,481 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,482 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,482 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,582 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,586 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,586 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,588 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,588 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,589 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,590 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,590 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,592 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,592 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c03e224[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,594 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,595 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,595 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,598 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,598 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,602 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,602 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,604 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,604 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,604 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,605 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,606 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,608 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,609 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,609 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,609 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,609 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 639 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419454632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 622 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419454635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,710 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,713 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,713 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,714 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,715 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,715 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,715 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,716 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,717 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,717 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,720 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,720 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50942374[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,722 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,723 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,723 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,723 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,725 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,725 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,727 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,728 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,728 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,733 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,733 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,735 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,737 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,737 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,737 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,738 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,738 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,744 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,768 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=2093, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/f817b05344294fec8d95dd783767742d 2023-04-13 20:56:34,838 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,841 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,841 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,843 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,843 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,843 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,843 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,845 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,845 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,845 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,845 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,845 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,848 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,848 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77098073[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,850 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,852 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,852 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,852 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,854 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,854 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,857 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,857 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,858 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,859 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,860 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,863 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,863 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,863 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,863 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,963 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:34,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:34,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:34,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:34,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:34,966 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:34,966 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:34,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:34,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,967 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,969 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,969 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,970 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:34,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:34,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:34,971 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:34,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:34,972 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:34,973 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,973 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,974 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,974 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:34,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ba5e02[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:34,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:34,976 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,978 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:34,978 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,978 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:34,978 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:34,979 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,979 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:34,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:34,982 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:34,982 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:34,982 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:34,983 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:34,983 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:34,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,985 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:34,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,986 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,987 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,990 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:34,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:34,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:34,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:34,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:34,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:34,990 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:34,991 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:34,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:34,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:34,991 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:34,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39670 deadline: 1681419454991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,994 DEBUG [Thread-4335] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:34,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:34,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 414 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39658 deadline: 1681419454992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:34,997 DEBUG [Thread-4333] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:35,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:35,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36431] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39700 deadline: 1681419455007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,010 DEBUG [Thread-4331] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., hostname=jenkins-hbase4.apache.org,36431,1681419217857, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-13 20:56:35,091 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,094 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,094 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,095 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,096 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,096 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,097 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:35,098 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,098 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:35,099 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,100 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,100 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:35,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:35,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35bb8a33[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:35,101 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:35,102 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,103 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:35,104 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:35,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,104 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,105 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,105 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:35,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:35,108 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:35,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:35,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:35,108 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:35,109 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:35,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,110 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,111 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,112 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,122 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,122 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:35,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:35,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,123 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:35,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,123 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:35,123 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:35,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:35,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 641 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39674 deadline: 1681419455137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-13 20:56:35,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36431] ipc.CallRunner(144): callId: 624 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39694 deadline: 1681419455143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,178 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=2093 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/7bcb26f1feeb454090a922a9fcd9fbf0 2023-04-13 20:56:35,223 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,227 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,227 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,228 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,228 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,229 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,230 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,230 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,230 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:35,230 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:35,230 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:35,232 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,232 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,232 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,232 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:35,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:35,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44458073[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:35,233 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:35,234 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,235 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,235 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:35,235 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:35,235 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:35,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,235 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:35,237 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:35,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,238 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:35,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:35,240 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:35,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:35,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:35,240 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:35,240 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:35,241 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,242 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,243 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:35,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:35,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:35,246 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,246 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:35,246 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:35,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,246 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,247 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,247 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,247 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,247 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,346 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,350 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,350 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,352 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,353 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,353 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,353 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,354 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:35,355 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:35,355 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:35,357 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,357 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,358 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:35,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f749a90[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:35,359 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:35,359 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,360 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:35,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:35,361 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:35,361 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,361 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:35,363 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:35,363 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,363 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,363 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:35,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:35,365 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:35,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:35,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:35,366 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:35,366 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:35,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,367 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,368 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,369 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,372 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:35,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:35,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,372 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,372 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:35,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,372 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,473 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,476 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,476 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,477 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,478 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,478 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,479 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:35,480 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:35,480 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:35,482 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,482 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,483 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:35,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bca9132[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:35,484 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:35,484 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,486 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:35,486 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:35,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,486 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:35,488 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:35,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,488 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:35,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:35,490 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:35,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:35,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:35,490 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:35,490 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:35,492 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,492 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,492 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,493 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,494 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,498 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,497 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,498 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:35,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:35,498 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:35,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,498 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:35,589 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=2093 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/b00dd31f490a46319f50fc597edcfe82 2023-04-13 20:56:35,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/f817b05344294fec8d95dd783767742d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d 2023-04-13 20:56:35,598 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,602 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,602 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d, entries=150, sequenceid=2093, filesize=25.4 K 2023-04-13 20:56:35,606 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/7bcb26f1feeb454090a922a9fcd9fbf0 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0 2023-04-13 20:56:35,610 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,610 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,611 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,612 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,614 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0, entries=150, sequenceid=2093, filesize=11.9 K 2023-04-13 20:56:35,614 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/b00dd31f490a46319f50fc597edcfe82 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82 2023-04-13 20:56:35,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:35,616 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:35,628 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:35,630 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,630 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82, entries=150, sequenceid=2093, filesize=11.9 K 2023-04-13 20:56:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,631 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:35,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c9786a[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:35,632 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:35,632 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.73 KB/210672, currentSize=107.34 KB/109920 for b19b9c1a7b3fa24b26a81e15015b427d in 1686ms, sequenceid=2093, compaction requested=true 2023-04-13 20:56:35,634 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:35,634 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:35,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:35,634 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:35,634 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:35,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:35,636 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,637 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138118 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:35,637 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/A is initiating minor compaction (all files) 2023-04-13 20:56:35,637 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/A in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,637 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d526228190814ec18711b4f06ba015c1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=134.9 K 2023-04-13 20:56:35,638 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting d526228190814ec18711b4f06ba015c1, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:35,638 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 61bc3b47e71448b3b684cb4e589195b2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1851, earliestPutTs=1681419392178 2023-04-13 20:56:35,638 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:35,638 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/B is initiating minor compaction (all files) 2023-04-13 20:56:35,639 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/B in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,639 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/561e8ad893dc4aa3bb3904f36fa378c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=57.9 K 2023-04-13 20:56:35,639 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 7b1623a7690841e495d5e559a1e028b6, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1869, earliestPutTs=1681419392815 2023-04-13 20:56:35,640 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 561e8ad893dc4aa3bb3904f36fa378c7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:35,640 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting 497f254a0a52434c880e5a32587b1284, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1876, earliestPutTs=1681419393448 2023-04-13 20:56:35,640 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting aee400c6c64240e98ac689a60efbc6a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1851, earliestPutTs=1681419392178 2023-04-13 20:56:35,640 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] compactions.Compactor(207): Compacting f817b05344294fec8d95dd783767742d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=2093, earliestPutTs=1681419393933 2023-04-13 20:56:35,640 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:35,641 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:35,641 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:35,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,641 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:35,642 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b31620e0cf164a7d904fae051c0c17be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1869, earliestPutTs=1681419392815 2023-04-13 20:56:35,642 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 0181c894aab3436ab5208b173511dbd3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1876, earliestPutTs=1681419393448 2023-04-13 20:56:35,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:35,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,642 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,642 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,643 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting 7bcb26f1feeb454090a922a9fcd9fbf0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=2093, earliestPutTs=1681419393933 2023-04-13 20:56:35,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:35,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:35,646 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:35,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:35,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:35,646 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:35,646 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:35,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:35,648 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:35,650 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,652 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:35,654 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,655 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:35,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:35,655 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:35,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:35,655 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:35,665 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#B#compaction#671 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:35,702 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#A#compaction#672 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:35,708 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/d718b3c5f39d40dab60a66affb760fcc as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/d718b3c5f39d40dab60a66affb760fcc 2023-04-13 20:56:35,713 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/B of b19b9c1a7b3fa24b26a81e15015b427d into d718b3c5f39d40dab60a66affb760fcc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:35,713 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:35,713 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/B, priority=11, startTime=1681419395634; duration=0sec 2023-04-13 20:56:35,713 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-13 20:56:35,713 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-13 20:56:35,714 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59338 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-13 20:56:35,715 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1912): b19b9c1a7b3fa24b26a81e15015b427d/C is initiating minor compaction (all files) 2023-04-13 20:56:35,715 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b19b9c1a7b3fa24b26a81e15015b427d/C in TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,715 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b643291046214144b5efc8ba76212171, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82] into tmpdir=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp, totalSize=57.9 K 2023-04-13 20:56:35,715 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b643291046214144b5efc8ba76212171, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1834, earliestPutTs=1681419391553 2023-04-13 20:56:35,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting dfe7bae2e57941d88b3b1ec9614d1061, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1851, earliestPutTs=1681419392178 2023-04-13 20:56:35,716 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting efe4c52a3e414048bc9209d5c79945a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1869, earliestPutTs=1681419392815 2023-04-13 20:56:35,717 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting afd8b0e8b250401a99ce223ce7e3b682, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1876, earliestPutTs=1681419393448 2023-04-13 20:56:35,717 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] compactions.Compactor(207): Compacting b00dd31f490a46319f50fc597edcfe82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=2093, earliestPutTs=1681419393933 2023-04-13 20:56:35,726 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] throttle.PressureAwareThroughputController(145): b19b9c1a7b3fa24b26a81e15015b427d#C#compaction#673 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-13 20:56:35,737 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/803f2a12bdb048eb8f480a17efe2d2f9 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/803f2a12bdb048eb8f480a17efe2d2f9 2023-04-13 20:56:35,741 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/C of b19b9c1a7b3fa24b26a81e15015b427d into 803f2a12bdb048eb8f480a17efe2d2f9(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-13 20:56:35,741 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:35,741 INFO [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/C, priority=11, startTime=1681419395635; duration=0sec 2023-04-13 20:56:35,742 DEBUG [RS:0;jenkins-hbase4:36431-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:35,755 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:35,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:35,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-13 20:56:35,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-13 20:56:35,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-13 20:56:35,758 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-13 20:56:35,758 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-13 20:56:35,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-13 20:56:35,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,759 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,763 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:35,763 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:35,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:35,764 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-13 20:56:35,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-13 20:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:35,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-13 20:56:35,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-13 20:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-13 20:56:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-13 20:56:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-13 20:56:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-13 20:56:35,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-13 20:56:35,766 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. started... 2023-04-13 20:56:35,766 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:35,766 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:35,785 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304137b98ed7b24db43e093356041bf463957 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b98ed7b24db43e093356041bf463957 2023-04-13 20:56:36,082 DEBUG [Thread-4342] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:54584 2023-04-13 20:56:36,082 DEBUG [Thread-4342] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,083 DEBUG [Thread-4346] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:54584 2023-04-13 20:56:36,083 DEBUG [Thread-4344] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:54584 2023-04-13 20:56:36,083 DEBUG [Thread-4344] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,083 DEBUG [Thread-4346] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,084 DEBUG [Thread-4348] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:54584 2023-04-13 20:56:36,084 DEBUG [Thread-4348] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,084 DEBUG [Thread-4350] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:54584 2023-04-13 20:56:36,084 DEBUG [Thread-4350] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36431] regionserver.HRegion(9158): Flush requested on b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:36,152 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. as already flushing 2023-04-13 20:56:36,152 DEBUG [Thread-4339] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:54584 2023-04-13 20:56:36,152 DEBUG [Thread-4339] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,156 DEBUG [Thread-4337] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:54584 2023-04-13 20:56:36,156 DEBUG [Thread-4337] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:36,194 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=2114, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/ff2197f2cc0844b29ff9f37ff014a72d 2023-04-13 20:56:36,602 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=2114 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/c368e64ee3934b7bacba4fb1657bbcbb 2023-04-13 20:56:36,929 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/8955cd26a8c24046bee4156728304f1c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8955cd26a8c24046bee4156728304f1c 2023-04-13 20:56:36,932 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in b19b9c1a7b3fa24b26a81e15015b427d/A of b19b9c1a7b3fa24b26a81e15015b427d into 8955cd26a8c24046bee4156728304f1c(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-13 20:56:36,932 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:36,932 INFO [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d., storeName=b19b9c1a7b3fa24b26a81e15015b427d/A, priority=11, startTime=1681419395634; duration=1sec 2023-04-13 20:56:36,933 DEBUG [RS:0;jenkins-hbase4:36431-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-13 20:56:37,010 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=2114 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/e621969be7c64c27912a78e84c8e6ae4 2023-04-13 20:56:37,013 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/ff2197f2cc0844b29ff9f37ff014a72d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/ff2197f2cc0844b29ff9f37ff014a72d 2023-04-13 20:56:37,016 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/ff2197f2cc0844b29ff9f37ff014a72d, entries=150, sequenceid=2114, filesize=25.4 K 2023-04-13 20:56:37,017 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/c368e64ee3934b7bacba4fb1657bbcbb as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/c368e64ee3934b7bacba4fb1657bbcbb 2023-04-13 20:56:37,019 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/c368e64ee3934b7bacba4fb1657bbcbb, entries=150, sequenceid=2114, filesize=11.9 K 2023-04-13 20:56:37,020 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/e621969be7c64c27912a78e84c8e6ae4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e621969be7c64c27912a78e84c8e6ae4 2023-04-13 20:56:37,023 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e621969be7c64c27912a78e84c8e6ae4, entries=150, sequenceid=2114, filesize=11.9 K 2023-04-13 20:56:37,023 INFO [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=13.42 KB/13740 for b19b9c1a7b3fa24b26a81e15015b427d in 1257ms, sequenceid=2114, compaction requested=false 2023-04-13 20:56:37,023 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:37,024 DEBUG [rs(jenkins-hbase4.apache.org,36431,1681419217857)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:37,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-13 20:56:37,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-13 20:56:37,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-13 20:56:37,024 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-13 20:56:37,025 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:37,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:37,025 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,026 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-13 20:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36431,1681419217857' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-13 20:56:37,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae72a58[Count = 0] remaining members to acquire global barrier 2023-04-13 20:56:37,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-13 20:56:37,027 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,029 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-13 20:56:37,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-13 20:56:37,029 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,36431,1681419217857' in zk 2023-04-13 20:56:37,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,029 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-13 20:56:37,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-13 20:56:37,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:37,030 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,030 DEBUG [member: 'jenkins-hbase4.apache.org,36431,1681419217857' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,36431,1681419217857': 2023-04-13 20:56:37,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36431,1681419217857' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-13 20:56:37,032 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-13 20:56:37,032 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-13 20:56:37,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-13 20:56:37,033 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-13 20:56:37,033 INFO [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-13 20:56:37,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-13 20:56:37,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,034 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-13 20:56:37,034 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:37,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-13 20:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,035 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-13 20:56:37,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-13 20:56:37,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-13 20:56:37,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,037 DEBUG [(jenkins-hbase4.apache.org,43163,1681419216653)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-13 20:56:37,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-13 20:56:37,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:37,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-13 20:56:37,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-13 20:56:37,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-13 20:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-13 20:56:37,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-13 20:56:37,037 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-13 20:56:37,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:37,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-13 20:56:37,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-13 20:56:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-13 20:56:37,038 DEBUG [Thread-4341] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-13 20:56:37,038 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-13 20:56:37,038 DEBUG [Thread-4341] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-13 20:56:37,138 DEBUG [Thread-4341] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-13 20:56:37,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-13 20:56:39,275 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-13 20:56:45,023 DEBUG [Thread-4335] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:54584 2023-04-13 20:56:45,023 DEBUG [Thread-4335] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:45,026 DEBUG [Thread-4331] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:54584 2023-04-13 20:56:45,027 DEBUG [Thread-4331] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:45,055 DEBUG [Thread-4333] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:54584 2023-04-13 20:56:45,055 DEBUG [Thread-4333] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 332 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 335 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 337 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 495 2023-04-13 20:56:45,055 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(394): wrote 514 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4645 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4489 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4485 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4632 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(398): read 4490 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:56:45,056 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:54584 2023-04-13 20:56:45,056 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:45,056 INFO [Listener at localhost/42015] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-13 20:56:45,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-13 20:56:45,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:45,059 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419405059"}]},"ts":"1681419405059"} 2023-04-13 20:56:45,060 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-13 20:56:45,062 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-13 20:56:45,062 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, UNASSIGN}] 2023-04-13 20:56:45,064 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, UNASSIGN 2023-04-13 20:56:45,064 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:45,064 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419405064"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681419405064"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681419405064"}]},"ts":"1681419405064"} 2023-04-13 20:56:45,065 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857}] 2023-04-13 20:56:45,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:45,216 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:45,216 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b19b9c1a7b3fa24b26a81e15015b427d, disabling compactions & flushes 2023-04-13 20:56:45,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. after waiting 0 ms 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:45,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b19b9c1a7b3fa24b26a81e15015b427d 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=A 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=B 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b19b9c1a7b3fa24b26a81e15015b427d, store=C 2023-04-13 20:56:45,217 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-13 20:56:45,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:45,628 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230413cad50417d5054fe7b0e885e126218f55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cad50417d5054fe7b0e885e126218f55 2023-04-13 20:56:45,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:46,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=2123, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/5905306382534758829a3509da641e7d 2023-04-13 20:56:46,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:46,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=2123 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/863252385d214809b291bcefd323a6f4 2023-04-13 20:56:46,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=2123 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/71db7e7c262c4d5fb1ba75e4b8854b6f 2023-04-13 20:56:46,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/A/5905306382534758829a3509da641e7d as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/5905306382534758829a3509da641e7d 2023-04-13 20:56:46,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/5905306382534758829a3509da641e7d, entries=150, sequenceid=2123, filesize=25.4 K 2023-04-13 20:56:46,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/B/863252385d214809b291bcefd323a6f4 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/863252385d214809b291bcefd323a6f4 2023-04-13 20:56:46,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/863252385d214809b291bcefd323a6f4, entries=150, sequenceid=2123, filesize=11.9 K 2023-04-13 20:56:46,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/.tmp/C/71db7e7c262c4d5fb1ba75e4b8854b6f as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71db7e7c262c4d5fb1ba75e4b8854b6f 2023-04-13 20:56:46,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71db7e7c262c4d5fb1ba75e4b8854b6f, entries=150, sequenceid=2123, filesize=11.9 K 2023-04-13 20:56:46,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for b19b9c1a7b3fa24b26a81e15015b427d in 1644ms, sequenceid=2123, compaction requested=true 2023-04-13 20:56:46,883 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/6091ee476c9d45d79f7dded201a3ce12, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/640e634cbf1448d3b6dbeb557850849b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/db893706695c440a9914753fe57a2be0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0de5ce0864a94f7bb3ecf87f31ace0bf, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e5c419fdd94f404ea174ddb8dd7058a8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d526228190814ec18711b4f06ba015c1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d] to archive 2023-04-13 20:56:46,884 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:46,885 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e9c8cfe753ef4a9bba9cd3c391f237a8 2023-04-13 20:56:46,886 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/9d40a97a82904e6c9bc79bc1fc38ddf8 2023-04-13 20:56:46,887 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/30a6b8c5b0b842528ba6c4490c460249 2023-04-13 20:56:46,888 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d8b15c89a4ef4025b68baedd5c1a1a36 2023-04-13 20:56:46,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/6091ee476c9d45d79f7dded201a3ce12 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/6091ee476c9d45d79f7dded201a3ce12 2023-04-13 20:56:46,889 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/c1f6908200c34ad38f006d8d4015fe6c 2023-04-13 20:56:46,890 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/331723f764cb464e856f4b60839b4f55 2023-04-13 20:56:46,891 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0a551e37095143b79f01b230b4dab5d7 2023-04-13 20:56:46,892 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/283133053d8c499596178526b22d77c4 2023-04-13 20:56:46,893 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/640e634cbf1448d3b6dbeb557850849b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/640e634cbf1448d3b6dbeb557850849b 2023-04-13 20:56:46,894 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/958cbcde9cfe49698476cdb15577af8d 2023-04-13 20:56:46,895 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/a51d31566bb14107be67fc17c65c16c7 2023-04-13 20:56:46,897 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/aa7208b9db00407caf9c142724d99192 2023-04-13 20:56:46,898 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/995c63b4db2d4ef29f78866f04ebead6 2023-04-13 20:56:46,899 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/db893706695c440a9914753fe57a2be0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/db893706695c440a9914753fe57a2be0 2023-04-13 20:56:46,900 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7602f9895dc24acdaf2d78be1c1d5653 2023-04-13 20:56:46,901 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/bab0397aa7854864967bb6334a4c5b54 2023-04-13 20:56:46,903 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/3372285868b9484791ada4b1ca116822 2023-04-13 20:56:46,904 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0de5ce0864a94f7bb3ecf87f31ace0bf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/0de5ce0864a94f7bb3ecf87f31ace0bf 2023-04-13 20:56:46,905 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/720c627ac111417bbeaceb40bb506790 2023-04-13 20:56:46,906 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/b04599fca5194f3aa70ee6b8c6a048f2 2023-04-13 20:56:46,909 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e5c419fdd94f404ea174ddb8dd7058a8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/e5c419fdd94f404ea174ddb8dd7058a8 2023-04-13 20:56:46,910 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8bdf6ece176e4b3eb1c66c2ae3709d79 2023-04-13 20:56:46,911 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/03127ff119da4ce9a648f4dcb6ac4606 2023-04-13 20:56:46,912 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/93f03849e14040949771325f52410f3e 2023-04-13 20:56:46,913 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d526228190814ec18711b4f06ba015c1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/d526228190814ec18711b4f06ba015c1 2023-04-13 20:56:46,914 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/61bc3b47e71448b3b684cb4e589195b2 2023-04-13 20:56:46,915 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/7b1623a7690841e495d5e559a1e028b6 2023-04-13 20:56:46,916 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/497f254a0a52434c880e5a32587b1284 2023-04-13 20:56:46,917 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/f817b05344294fec8d95dd783767742d 2023-04-13 20:56:46,920 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/077b6e6ff527442186b98ab06e8d9281, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ed52215047394c4eb34168a16fc57730, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/83af0b8771b749e990087b933def1345, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0c0b148d03b84db58c11b8cea3c3593e, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3351a0e5715f47f9bb8e1c6b547f29fe, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/561e8ad893dc4aa3bb3904f36fa378c7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0] to archive 2023-04-13 20:56:46,921 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:46,922 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0dccd0fc91714de0a0202d1f6f2e1970 2023-04-13 20:56:46,923 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/1d061409580446fba5e69aa9642a5819 2023-04-13 20:56:46,924 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/78df554aad6c42f9b4b7feb577aeba08 2023-04-13 20:56:46,925 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/21f86875e73640e5b325364d9ab0fa1a 2023-04-13 20:56:46,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/077b6e6ff527442186b98ab06e8d9281 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/077b6e6ff527442186b98ab06e8d9281 2023-04-13 20:56:46,926 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5aea6bdc13c1461e93aa66807286ea55 2023-04-13 20:56:46,927 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/612461aab0ed48d385a41a3a1ce693bd 2023-04-13 20:56:46,928 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7029a2a7790c4edcb8241f9194082153 2023-04-13 20:56:46,929 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ed52215047394c4eb34168a16fc57730 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ed52215047394c4eb34168a16fc57730 2023-04-13 20:56:46,930 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/9d83033ebea548bfa10d811d7d172ec5 2023-04-13 20:56:46,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/413647e8933b423b807042187f2d411b 2023-04-13 20:56:46,931 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/eb7bf54fcd0a44a59c29b669999bcad9 2023-04-13 20:56:46,932 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/122a52bcaad342909d833c81dcf5e6e5 2023-04-13 20:56:46,933 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/83af0b8771b749e990087b933def1345 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/83af0b8771b749e990087b933def1345 2023-04-13 20:56:46,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/5a8338b836a44243a942036056d58d32 2023-04-13 20:56:46,934 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3e6b9bcb01ec4811b5b8d3fba651498e 2023-04-13 20:56:46,935 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/cf6ea963d2cd4e3899afc426320e2910 2023-04-13 20:56:46,936 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0c0b148d03b84db58c11b8cea3c3593e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0c0b148d03b84db58c11b8cea3c3593e 2023-04-13 20:56:46,937 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/762052d8893142ed83034855a1ec2dc8 2023-04-13 20:56:46,938 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/fef45f65ca7a48c3b798a3fb87e1c8a2 2023-04-13 20:56:46,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3351a0e5715f47f9bb8e1c6b547f29fe to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/3351a0e5715f47f9bb8e1c6b547f29fe 2023-04-13 20:56:46,939 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/bb0da5eec92f48a896a95989daebf184 2023-04-13 20:56:46,940 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/8df63e7ffe38467abb2c2bbd8ea85ba2 2023-04-13 20:56:46,941 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/f3455732d18a428aabc3ebaa1df76fd0 2023-04-13 20:56:46,943 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/561e8ad893dc4aa3bb3904f36fa378c7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/561e8ad893dc4aa3bb3904f36fa378c7 2023-04-13 20:56:46,944 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/ad73cef1e42647468497955942cb03d3 2023-04-13 20:56:46,945 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/aee400c6c64240e98ac689a60efbc6a2 2023-04-13 20:56:46,946 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/b31620e0cf164a7d904fae051c0c17be 2023-04-13 20:56:46,947 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/0181c894aab3436ab5208b173511dbd3 2023-04-13 20:56:46,948 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/7bcb26f1feeb454090a922a9fcd9fbf0 2023-04-13 20:56:46,951 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/978b20e090104056990883adb90ab67d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/5376e7e778a54edbbe9d55b8dde09559, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e00d7fe8c21c46b9be272d9b47164c11, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0c597546fe2e4f16b0199fa62170e6c4, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0cdb9d77bf464339ab93907a7e790b79, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b643291046214144b5efc8ba76212171, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82] to archive 2023-04-13 20:56:46,952 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-13 20:56:46,953 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e3032d7db71743bea4020829595b2fd8 2023-04-13 20:56:46,954 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/56f141414b934f3db87340410b1aed3d 2023-04-13 20:56:46,955 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b2c0f68f60ac401d95831c18e97fe8c9 2023-04-13 20:56:46,956 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/577017a3bd9a42829e24a12abe05c0a5 2023-04-13 20:56:46,957 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4c6b3203e82442c9bf22909977e6b8b0 2023-04-13 20:56:46,958 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/978b20e090104056990883adb90ab67d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/978b20e090104056990883adb90ab67d 2023-04-13 20:56:46,959 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8b2e379ba7af438db3851415d9644a72 2023-04-13 20:56:46,960 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/8ca75e670e444cfe924b9302f4618077 2023-04-13 20:56:46,961 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/5376e7e778a54edbbe9d55b8dde09559 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/5376e7e778a54edbbe9d55b8dde09559 2023-04-13 20:56:46,962 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/1887f990f0e24928ae370f3d1e05d492 2023-04-13 20:56:46,963 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/cd4377ad2a274ed1af9f87eca590bd89 2023-04-13 20:56:46,965 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/4e9b213a3d454762af44ac1abc85dfa2 2023-04-13 20:56:46,966 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/53d5a4f064c9444ebbc212e47dccb8ab 2023-04-13 20:56:46,967 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e00d7fe8c21c46b9be272d9b47164c11 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e00d7fe8c21c46b9be272d9b47164c11 2023-04-13 20:56:46,968 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/127b98ce00854cd685777703ec93e5bd 2023-04-13 20:56:46,969 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71578e2a8e974a82b79c657f25fdbed1 2023-04-13 20:56:46,970 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/6f5c4ef7a7654235be443a2498cf4f0a 2023-04-13 20:56:46,971 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0c597546fe2e4f16b0199fa62170e6c4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0c597546fe2e4f16b0199fa62170e6c4 2023-04-13 20:56:46,972 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/db81659f487f497196b3a8647f13cd29 2023-04-13 20:56:46,973 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/f64a053975ec4caca22eaabe481222df 2023-04-13 20:56:46,974 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0cdb9d77bf464339ab93907a7e790b79 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/0cdb9d77bf464339ab93907a7e790b79 2023-04-13 20:56:46,975 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/674a880b766349fdaf64d15f1ff0ad7a 2023-04-13 20:56:46,976 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/560df7fd06614a908747a2f946f00b2d 2023-04-13 20:56:46,977 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/37f2a9f0fa2449daa86856ce49502737 2023-04-13 20:56:46,978 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b643291046214144b5efc8ba76212171 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b643291046214144b5efc8ba76212171 2023-04-13 20:56:46,979 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/54a71353e4ab4a239eb56f36646d62d7 2023-04-13 20:56:46,980 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/dfe7bae2e57941d88b3b1ec9614d1061 2023-04-13 20:56:46,981 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/efe4c52a3e414048bc9209d5c79945a7 2023-04-13 20:56:46,982 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/afd8b0e8b250401a99ce223ce7e3b682 2023-04-13 20:56:46,983 DEBUG [StoreCloser-TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/b00dd31f490a46319f50fc597edcfe82 2023-04-13 20:56:46,987 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits/2126.seqid, newMaxSeqId=2126, maxSeqId=4 2023-04-13 20:56:46,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d. 2023-04-13 20:56:46,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b19b9c1a7b3fa24b26a81e15015b427d: 2023-04-13 20:56:46,990 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:46,990 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=b19b9c1a7b3fa24b26a81e15015b427d, regionState=CLOSED 2023-04-13 20:56:46,990 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1681419406990"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681419406990"}]},"ts":"1681419406990"} 2023-04-13 20:56:46,992 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-13 20:56:46,992 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure b19b9c1a7b3fa24b26a81e15015b427d, server=jenkins-hbase4.apache.org,36431,1681419217857 in 1.9260 sec 2023-04-13 20:56:46,993 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-13 20:56:46,993 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b19b9c1a7b3fa24b26a81e15015b427d, UNASSIGN in 1.9300 sec 2023-04-13 20:56:46,993 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681419406993"}]},"ts":"1681419406993"} 2023-04-13 20:56:46,994 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-13 20:56:46,995 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-13 20:56:46,997 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9390 sec 2023-04-13 20:56:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-13 20:56:47,164 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-13 20:56:47,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-13 20:56:47,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,167 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,167 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 20:56:47,171 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:47,173 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C, FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits] 2023-04-13 20:56:47,175 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/5905306382534758829a3509da641e7d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/5905306382534758829a3509da641e7d 2023-04-13 20:56:47,176 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8955cd26a8c24046bee4156728304f1c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/8955cd26a8c24046bee4156728304f1c 2023-04-13 20:56:47,177 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/ff2197f2cc0844b29ff9f37ff014a72d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/A/ff2197f2cc0844b29ff9f37ff014a72d 2023-04-13 20:56:47,179 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/863252385d214809b291bcefd323a6f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/863252385d214809b291bcefd323a6f4 2023-04-13 20:56:47,180 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/c368e64ee3934b7bacba4fb1657bbcbb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/c368e64ee3934b7bacba4fb1657bbcbb 2023-04-13 20:56:47,180 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/d718b3c5f39d40dab60a66affb760fcc to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/B/d718b3c5f39d40dab60a66affb760fcc 2023-04-13 20:56:47,182 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71db7e7c262c4d5fb1ba75e4b8854b6f to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/71db7e7c262c4d5fb1ba75e4b8854b6f 2023-04-13 20:56:47,183 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/803f2a12bdb048eb8f480a17efe2d2f9 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/803f2a12bdb048eb8f480a17efe2d2f9 2023-04-13 20:56:47,184 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e621969be7c64c27912a78e84c8e6ae4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/C/e621969be7c64c27912a78e84c8e6ae4 2023-04-13 20:56:47,186 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits/2126.seqid to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d/recovered.edits/2126.seqid 2023-04-13 20:56:47,186 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/.tmp/data/default/TestAcidGuarantees/b19b9c1a7b3fa24b26a81e15015b427d 2023-04-13 20:56:47,186 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-13 20:56:47,187 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:56:47,187 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-13 20:56:47,190 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130034ff8e655f446985f261a8f1e9aa06 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304130034ff8e655f446985f261a8f1e9aa06 2023-04-13 20:56:47,191 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306049ba412be45918449ac6f3f822c8c to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041306049ba412be45918449ac6f3f822c8c 2023-04-13 20:56:47,192 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413158a5b37ea454237875013419a0089ac to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413158a5b37ea454237875013419a0089ac 2023-04-13 20:56:47,193 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131cc53684568a43ea8c24c267a8e04316 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131cc53684568a43ea8c24c267a8e04316 2023-04-13 20:56:47,193 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131febbd98b23a44c2bc32b3e96cf6c8cb to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304131febbd98b23a44c2bc32b3e96cf6c8cb 2023-04-13 20:56:47,194 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041321ca15f34a5f4537a5e9cc269332df3e to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041321ca15f34a5f4537a5e9cc269332df3e 2023-04-13 20:56:47,195 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132cb83587a6e64463ae801d97821cab56 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304132cb83587a6e64463ae801d97821cab56 2023-04-13 20:56:47,196 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413303229c0c7d148838d39c2363131c69d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413303229c0c7d148838d39c2363131c69d 2023-04-13 20:56:47,197 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041333cf87f925dd4dcd8476ad3393274437 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041333cf87f925dd4dcd8476ad3393274437 2023-04-13 20:56:47,198 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413357233018ff44d8393fddec05b60a4d4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413357233018ff44d8393fddec05b60a4d4 2023-04-13 20:56:47,198 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413397d2d09d5744e81a6a72872256181f4 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413397d2d09d5744e81a6a72872256181f4 2023-04-13 20:56:47,199 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133d461a96a52d4cd3876bfcc8e54b90af to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304133d461a96a52d4cd3876bfcc8e54b90af 2023-04-13 20:56:47,200 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a01fb122ae4d79a9e1161fdd9f1a55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041356a01fb122ae4d79a9e1161fdd9f1a55 2023-04-13 20:56:47,201 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c4a4ceef68642b28af761fd69a47c2d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304135c4a4ceef68642b28af761fd69a47c2d 2023-04-13 20:56:47,202 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041369247f18b29849c89a7c34de9234966a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023041369247f18b29849c89a7c34de9234966a 2023-04-13 20:56:47,203 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e3bdc2406c94fb4b7aaac235aaab520 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304136e3bdc2406c94fb4b7aaac235aaab520 2023-04-13 20:56:47,216 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b98ed7b24db43e093356041bf463957 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304137b98ed7b24db43e093356041bf463957 2023-04-13 20:56:47,217 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138cc29a9265fc49d1b4a59ceec6ea471b to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304138cc29a9265fc49d1b4a59ceec6ea471b 2023-04-13 20:56:47,218 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cad50417d5054fe7b0e885e126218f55 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413cad50417d5054fe7b0e885e126218f55 2023-04-13 20:56:47,219 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d550291bfd5548ca801ee06ce129ee1a to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d550291bfd5548ca801ee06ce129ee1a 2023-04-13 20:56:47,219 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d82150c7d3734e478474a4bde3e3de13 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d82150c7d3734e478474a4bde3e3de13 2023-04-13 20:56:47,220 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d93ceb39405a4e228e195cb8cc47863d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413d93ceb39405a4e228e195cb8cc47863d 2023-04-13 20:56:47,221 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e7b8e84d9a6d4be19be2e2199612faaa to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413e7b8e84d9a6d4be19be2e2199612faaa 2023-04-13 20:56:47,222 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f10b79ae413d4ddf96b46e4454c134cf to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f10b79ae413d4ddf96b46e4454c134cf 2023-04-13 20:56:47,223 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f470096c3e584bb89c3e3e7dacd4292d to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413f470096c3e584bb89c3e3e7dacd4292d 2023-04-13 20:56:47,224 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fbc524bd0685434b9a4f892c0c499421 to hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230413fbc524bd0685434b9a4f892c0c499421 2023-04-13 20:56:47,224 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-13 20:56:47,225 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,227 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-13 20:56:47,228 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-13 20:56:47,229 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,229 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-13 20:56:47,229 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681419407229"}]},"ts":"9223372036854775807"} 2023-04-13 20:56:47,231 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-13 20:56:47,231 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b19b9c1a7b3fa24b26a81e15015b427d, NAME => 'TestAcidGuarantees,,1681419373986.b19b9c1a7b3fa24b26a81e15015b427d.', STARTKEY => '', ENDKEY => ''}] 2023-04-13 20:56:47,231 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-13 20:56:47,231 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681419407231"}]},"ts":"9223372036854775807"} 2023-04-13 20:56:47,232 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-13 20:56:47,233 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-13 20:56:47,234 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-04-13 20:56:47,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43163] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-13 20:56:47,268 INFO [Listener at localhost/42015] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-13 20:56:47,279 INFO [Listener at localhost/42015] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 228), OpenFileDescriptor=519 (was 589), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=413 (was 422), ProcessCount=173 (was 173), AvailableMemoryMB=894 (was 887) - AvailableMemoryMB LEAK? - 2023-04-13 20:56:47,280 INFO [Listener at localhost/42015] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-13 20:56:47,280 INFO [Listener at localhost/42015] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-13 20:56:47,280 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x373f9c99 to 127.0.0.1:54584 2023-04-13 20:56:47,280 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:47,281 DEBUG [Listener at localhost/42015] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-13 20:56:47,281 DEBUG [Listener at localhost/42015] util.JVMClusterUtil(257): Found active master hash=1464154429, stopped=false 2023-04-13 20:56:47,281 INFO [Listener at localhost/42015] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:56:47,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 20:56:47,282 INFO [Listener at localhost/42015] procedure2.ProcedureExecutor(629): Stopping 2023-04-13 20:56:47,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:56:47,282 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-13 20:56:47,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 20:56:47,283 DEBUG [Listener at localhost/42015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5ded93 to 127.0.0.1:54584 2023-04-13 20:56:47,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-13 20:56:47,283 DEBUG [Listener at localhost/42015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:47,283 INFO [Listener at localhost/42015] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36431,1681419217857' ***** 2023-04-13 20:56:47,283 INFO [Listener at localhost/42015] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-13 20:56:47,283 INFO [RS:0;jenkins-hbase4:36431] regionserver.HeapMemoryManager(220): Stopping 2023-04-13 20:56:47,283 INFO [RS:0;jenkins-hbase4:36431] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-13 20:56:47,284 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-13 20:56:47,284 INFO [RS:0;jenkins-hbase4:36431] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-13 20:56:47,284 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(3304): Received CLOSE for 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:47,285 DEBUG [RS:0;jenkins-hbase4:36431] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c4735ad to 127.0.0.1:54584 2023-04-13 20:56:47,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 33a8ac462e5249c56ee9b85ce2d17674, disabling compactions & flushes 2023-04-13 20:56:47,285 DEBUG [RS:0;jenkins-hbase4:36431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:47,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:56:47,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-13 20:56:47,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. after waiting 0 ms 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-13 20:56:47,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:56:47,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 33a8ac462e5249c56ee9b85ce2d17674 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-13 20:56:47,285 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-13 20:56:47,286 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1479): Online Regions={33a8ac462e5249c56ee9b85ce2d17674=hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674., 1588230740=hbase:meta,,1.1588230740} 2023-04-13 20:56:47,286 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-13 20:56:47,286 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-13 20:56:47,286 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-13 20:56:47,286 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-13 20:56:47,286 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-13 20:56:47,286 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-13 20:56:47,288 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1505): Waiting on 1588230740, 33a8ac462e5249c56ee9b85ce2d17674 2023-04-13 20:56:47,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/.tmp/info/5f4aea8f62af4ea1939f9900edfc272c 2023-04-13 20:56:47,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/.tmp/info/5f4aea8f62af4ea1939f9900edfc272c as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/info/5f4aea8f62af4ea1939f9900edfc272c 2023-04-13 20:56:47,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/info/5f4aea8f62af4ea1939f9900edfc272c, entries=2, sequenceid=6, filesize=4.8 K 2023-04-13 20:56:47,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 33a8ac462e5249c56ee9b85ce2d17674 in 22ms, sequenceid=6, compaction requested=false 2023-04-13 20:56:47,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/namespace/33a8ac462e5249c56ee9b85ce2d17674/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-13 20:56:47,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:56:47,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 33a8ac462e5249c56ee9b85ce2d17674: 2023-04-13 20:56:47,312 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681419219779.33a8ac462e5249c56ee9b85ce2d17674. 2023-04-13 20:56:47,334 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-13 20:56:47,488 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 20:56:47,688 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 20:56:47,697 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/info/6baaf4632ce54235a60016ddae4b9184 2023-04-13 20:56:47,702 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6baaf4632ce54235a60016ddae4b9184 2023-04-13 20:56:47,707 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/rep_barrier/45a0c42f6dc0481fa6cd23bdde8eb886 2023-04-13 20:56:47,710 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45a0c42f6dc0481fa6cd23bdde8eb886 2023-04-13 20:56:47,888 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 20:56:48,089 DEBUG [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-13 20:56:48,115 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/table/abebcbd82efe489ba2f712f56e2e8ec3 2023-04-13 20:56:48,118 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for abebcbd82efe489ba2f712f56e2e8ec3 2023-04-13 20:56:48,119 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/info/6baaf4632ce54235a60016ddae4b9184 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/info/6baaf4632ce54235a60016ddae4b9184 2023-04-13 20:56:48,122 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 6baaf4632ce54235a60016ddae4b9184 2023-04-13 20:56:48,122 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/info/6baaf4632ce54235a60016ddae4b9184, entries=22, sequenceid=93, filesize=7.1 K 2023-04-13 20:56:48,123 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/rep_barrier/45a0c42f6dc0481fa6cd23bdde8eb886 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/rep_barrier/45a0c42f6dc0481fa6cd23bdde8eb886 2023-04-13 20:56:48,126 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45a0c42f6dc0481fa6cd23bdde8eb886 2023-04-13 20:56:48,126 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/rep_barrier/45a0c42f6dc0481fa6cd23bdde8eb886, entries=6, sequenceid=93, filesize=5.5 K 2023-04-13 20:56:48,126 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/.tmp/table/abebcbd82efe489ba2f712f56e2e8ec3 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/table/abebcbd82efe489ba2f712f56e2e8ec3 2023-04-13 20:56:48,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for abebcbd82efe489ba2f712f56e2e8ec3 2023-04-13 20:56:48,129 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/table/abebcbd82efe489ba2f712f56e2e8ec3, entries=9, sequenceid=93, filesize=5.5 K 2023-04-13 20:56:48,129 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 843ms, sequenceid=93, compaction requested=false 2023-04-13 20:56:48,136 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-13 20:56:48,137 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-13 20:56:48,138 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-13 20:56:48,138 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-13 20:56:48,138 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-13 20:56:48,177 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-13 20:56:48,177 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-13 20:56:48,289 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36431,1681419217857; all regions closed. 2023-04-13 20:56:48,297 DEBUG [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/oldWALs 2023-04-13 20:56:48,297 INFO [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36431%2C1681419217857.meta:.meta(num 1681419219498) 2023-04-13 20:56:48,301 DEBUG [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/oldWALs 2023-04-13 20:56:48,301 INFO [RS:0;jenkins-hbase4:36431] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36431%2C1681419217857:(num 1681419219344) 2023-04-13 20:56:48,302 DEBUG [RS:0;jenkins-hbase4:36431] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:48,302 INFO [RS:0;jenkins-hbase4:36431] regionserver.LeaseManager(133): Closed leases 2023-04-13 20:56:48,302 INFO [RS:0;jenkins-hbase4:36431] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-13 20:56:48,302 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 20:56:48,304 INFO [RS:0;jenkins-hbase4:36431] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36431 2023-04-13 20:56:48,309 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36431,1681419217857 2023-04-13 20:56:48,309 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 20:56:48,310 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2af18af2 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2745] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,310 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-13 20:56:48,310 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7cedfbf4 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2745] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,310 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36431,1681419217857] 2023-04-13 20:56:48,312 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36431,1681419217857; numProcessing=1 2023-04-13 20:56:48,313 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36431,1681419217857 already deleted, retry=false 2023-04-13 20:56:48,313 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36431,1681419217857 expired; onlineServers=0 2023-04-13 20:56:48,313 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43163,1681419216653' ***** 2023-04-13 20:56:48,313 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-13 20:56:48,313 DEBUG [M:0;jenkins-hbase4:43163] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16783306, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-13 20:56:48,313 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43163,1681419216653 2023-04-13 20:56:48,313 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43163,1681419216653; all regions closed. 2023-04-13 20:56:48,313 DEBUG [M:0;jenkins-hbase4:43163] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-13 20:56:48,314 DEBUG [M:0;jenkins-hbase4:43163] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-13 20:56:48,314 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-13 20:56:48,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681419218987] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681419218987,5,FailOnTimeoutGroup] 2023-04-13 20:56:48,314 DEBUG [M:0;jenkins-hbase4:43163] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-13 20:56:48,314 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681419218987] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681419218987,5,FailOnTimeoutGroup] 2023-04-13 20:56:48,314 INFO [M:0;jenkins-hbase4:43163] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-13 20:56:48,314 INFO [M:0;jenkins-hbase4:43163] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-13 20:56:48,314 INFO [M:0;jenkins-hbase4:43163] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-13 20:56:48,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-13 20:56:48,315 DEBUG [M:0;jenkins-hbase4:43163] master.HMaster(1512): Stopping service threads 2023-04-13 20:56:48,315 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-13 20:56:48,315 INFO [M:0;jenkins-hbase4:43163] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-13 20:56:48,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-13 20:56:48,315 ERROR [M:0;jenkins-hbase4:43163] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (249600520) connection to localhost/127.0.0.1:39631 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-13 20:56:48,316 INFO [M:0;jenkins-hbase4:43163] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-13 20:56:48,316 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-13 20:56:48,316 DEBUG [M:0;jenkins-hbase4:43163] zookeeper.ZKUtil(398): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-13 20:56:48,316 WARN [M:0;jenkins-hbase4:43163] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-13 20:56:48,316 INFO [M:0;jenkins-hbase4:43163] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-13 20:56:48,317 INFO [M:0;jenkins-hbase4:43163] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-13 20:56:48,317 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-13 20:56:48,317 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:56:48,317 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:56:48,317 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-13 20:56:48,317 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:56:48,317 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.04 KB heapSize=393.72 KB 2023-04-13 20:56:48,333 INFO [M:0;jenkins-hbase4:43163] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.04 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9e9910925a4f4a9eb2ac07ca67468b40 2023-04-13 20:56:48,337 INFO [M:0;jenkins-hbase4:43163] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9e9910925a4f4a9eb2ac07ca67468b40 2023-04-13 20:56:48,337 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/9e9910925a4f4a9eb2ac07ca67468b40 as hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9e9910925a4f4a9eb2ac07ca67468b40 2023-04-13 20:56:48,340 INFO [M:0;jenkins-hbase4:43163] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9e9910925a4f4a9eb2ac07ca67468b40 2023-04-13 20:56:48,340 INFO [M:0;jenkins-hbase4:43163] regionserver.HStore(1080): Added hdfs://localhost:39631/user/jenkins/test-data/c3e9af6e-38c2-58be-cc83-11f0adfb8fbe/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/9e9910925a4f4a9eb2ac07ca67468b40, entries=65, sequenceid=578, filesize=24.7 K 2023-04-13 20:56:48,341 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegion(2948): Finished flush of dataSize ~345.04 KB/353321, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 24ms, sequenceid=578, compaction requested=false 2023-04-13 20:56:48,342 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-13 20:56:48,342 DEBUG [M:0;jenkins-hbase4:43163] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-13 20:56:48,345 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-13 20:56:48,345 INFO [M:0;jenkins-hbase4:43163] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-13 20:56:48,346 INFO [M:0;jenkins-hbase4:43163] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43163 2023-04-13 20:56:48,353 DEBUG [M:0;jenkins-hbase4:43163] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43163,1681419216653 already deleted, retry=false 2023-04-13 20:56:48,412 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 20:56:48,412 INFO [RS:0;jenkins-hbase4:36431] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36431,1681419217857; zookeeper connection closed. 2023-04-13 20:56:48,412 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4266d03 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2745] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,413 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): regionserver:36431-0x1017f4d0d830001, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 20:56:48,413 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@e4226a5] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@e4226a5 2023-04-13 20:56:48,413 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@18c20755 rejected from java.util.concurrent.ThreadPoolExecutor@2ece0b74[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2745] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,414 INFO [Listener at localhost/42015] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-13 20:56:48,512 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 20:56:48,512 INFO [M:0;jenkins-hbase4:43163] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43163,1681419216653; zookeeper connection closed. 2023-04-13 20:56:48,512 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ac2afcf rejected from java.util.concurrent.ThreadPoolExecutor@7a2e13aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4585] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,513 DEBUG [Listener at localhost/42015-EventThread] zookeeper.ZKWatcher(600): master:43163-0x1017f4d0d830000, quorum=127.0.0.1:54584, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-13 20:56:48,513 ERROR [Listener at localhost/42015-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5490510e rejected from java.util.concurrent.ThreadPoolExecutor@7a2e13aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4585] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-13 20:56:48,514 WARN [Listener at localhost/42015] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-13 20:56:48,518 INFO [Listener at localhost/42015] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 20:56:48,626 WARN [BP-1610104274-172.31.14.131-1681419213238 heartbeating to localhost/127.0.0.1:39631] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-13 20:56:48,626 WARN [BP-1610104274-172.31.14.131-1681419213238 heartbeating to localhost/127.0.0.1:39631] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1610104274-172.31.14.131-1681419213238 (Datanode Uuid 1651cc58-dcf5-4483-aee0-3033ee4713c7) service to localhost/127.0.0.1:39631 2023-04-13 20:56:48,629 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582/dfs/data/data1/current/BP-1610104274-172.31.14.131-1681419213238] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 20:56:48,629 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/9e2e8a9c-a5de-bab8-7b81-c89543558a24/cluster_a2a65a09-667a-b7c6-2cca-9db46288b582/dfs/data/data2/current/BP-1610104274-172.31.14.131-1681419213238] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-13 20:56:48,665 INFO [Listener at localhost/42015] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-13 20:56:48,840 INFO [Listener at localhost/42015] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-13 20:56:49,037 INFO [Listener at localhost/42015] hbase.HBaseTestingUtility(1293): Minicluster is down